Skip to Main Content

Benchmarking Tools for Evaluating Robotic Assembly of Small Parts

Note: The RSS 2020 conference will be held virtually during the originally scheduled dates using the PheedLoop platform with workshops and other speaking sessions using Zoom teleconferencing. You can access the virtual conference materials on the RSS 2020 PheedLoop page.

Half-Day Workshop at Robotics: Science and Systems (RSS) July 13, 2020

Robot manipulating a gear on the NIST-ATB #1

Robot inserting a gear on the NIST-ATB #1

Workshop Materials, Presentations, and Papers 

All materials for the workshop including presentation files, submitted papers, and recordings of workshop sessions are available for download via a publicly accessible Google Drive folder.
Videos of all presentations and discussion can be watched on YouTube


Benchmarking tools have advanced the research and development of robotic manipulation capabilities, particularly for grasping (e.g., YCB Object Set). As robot capabilities progress and domains like manufacturing become populated with robot systems, performance artifacts for more advanced capabilities are needed. This workshop aims to promote the usage of a performance artifact for benchmarking robotic manipulation and assembly capabilities. The NIST Assembly Task Boards (NIST-ATB) are designed to quantify a robot system’s grasping, manipulation, and perception capabilities when used in small parts assembly operations. NIST-ATB #1 requires competencies including peg insertions, gear meshing, electrical connector insertions, and nut threading. It has been used in robotic manipulation competitions (e.g., IROS 2017 and 2019) and is currently being proliferated throughout the research community. Workshop content includes presentations of robotics research that utilizes the NIST-ATB for benchmarking purposes, such as the development of force sensing capabilities for insertion or the design of dexterous end-effectors to manipulate small objects. Related benchmarking and evaluation efforts involving the NIST-ATB will also be covered, as well as a discussion of the current and future plans for the benchmarking tool.

Submissions are elicited for extended abstracts of ongoing or proposed research using the NIST-ATB #1 benchmarking tool, which will be provided to authors of accepted contributions for free (a $450 value). Task boards can be shipped to your institution (if outside of the US, recipient may have to pay import duties/taxes upon delivery which can range between $50-150) or the task board can be picked up at the workshop. The goal of this workshop is to spur the development of advanced robotic capabilities and begin building a user community around robotic assembly research.

See the NIST website for more information on the assembly task boards. A paper was also recently published in the IEEE RA-L Special Issue on Benchmarking Protocols for Robotic Manipulation, Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems, that provides an overview of the NIST assembly task boards. The paper is freely available as open access.


  • Adam Norton, University of Massachusetts Lowell
  • Holly Yanco, University of Massachusetts Lowell
  • Joseph Falco, National Institute of Standards Technology (NIST)
  • Kenneth Kimble, National Institute of Standards and Technology (NIST)


Extended abstracts (1-2 pages, following RSS style guidelines) are being solicited to cover research towards robotic assembly and manufacturing, which may include end-to-end solutions and/or development of relevant components including vision algorithms, tactile sensing, end-effector design, and teaching/programming techniques. All contributions should cover ongoing or proposed research that will utilize the NIST-ATB #1 as part of their experimentation plan.

We are requesting 1-2 page extended abstracts using the RSS 2020 format to be submitted via e-mail to: Anonymization is not required. When emailing the paper, please include the title, author list, affiliations, and e-mail addresses in the body of the email. All submissions will be reviewed and authors of accepted papers will be asked to give a 10 minute talk at the workshop. At least one author of each accepted submission must register for the workshop. Authors of accepted submissions will be given a NIST-ATB #1 kit for free that will be shipped to them (if outside of the US, recipient may have to pay import duties/taxes upon delivery which can range between $50-150). Task boards may not be shipped to authors prior to the workshop due to COVID-19 restrictions, but the workshop organizers will ensure that all authors receive task boards once we are physically able to fabricate and ship them.

Authors of accepted submissions will be offered the option of having their papers uploaded to a workshop-specific archive on Inclusion in this archive will not be mandatory since it may create problems for authors who wish to submit follow-on work to venues with strict prior publication rules. Accepted authors will also be asked for shipping information to receive their task board.

  • March 6, 2020: Call for submissions open
  • June 19, 2020 23:59 Anywhere on Earth (AoE): Submission deadline for extended abstracts
  • June 29, 2020: Notification of acceptance for workshop submissions

All questions regarding the workshop can be e-mailed to:


Invited speakers use the NIST-ATB in their current research, either to benchmark the performance of robotic solutions they have developed (e.g., perception algorithms, end effector design) or other uses of the tool in evaluation methods. Accepted extended abstracts will present either ongoing or proposed research that will use the NIST-ATB.

  • Kenneth Kimble, National Institute of Standards and Technology (NIST): Benchmarking Protocols for Evaluating Small Parts Robotic Assembly Systems (organizer); watch on YouTube
  • James Watson and Siddhant Keshkar, University of Colorado, Boulder: Test of Assembly Algorithms with an Industry-Inspired Benchmark (invited); watch on YouTube
  • Adam Norton, University of Massachusetts Lowell: Test and Evaluation of Robot Capabilities in Industrial Environments (organizer); watch on YouTube
  • Juan Aparicio, Siemens: Developing Advanced Robotic Capabilities in Manufacturing Using Test and Evaluation Artifacts (invited); watch on YouTube
  • Cindy Grimm, Oregon State University: Automating Grasp Testing and Improving Ground Truth Estimations Using Sensorized Objects (invited); watch on YouTube
  • Alessandro Carfi, University of Genoa: From Human Demonstrations to Robot Manipulation: a Dataset for the NIST-ATB #1 Benchmark (submitted); watch on YouTube
  • Joao Silverio, Idiap Research Institute, Switzerland; Picking up and assembling watch components: an ongoing challenge for collaborative robots (submitted); watch on YouTube
  • Gal Gorjup, The University of Auckland, New Zealand; Combining Vision and CAD Based Localization for Efficient Programming of Assembly Tasks (submitted); watch on YouTube

Schedule (all times EST)

  • 10 a.m. - Welcome and overview
  • 10:15 a.m. - Talk 1: Kenneth Kimble, NIST
  • 10:35 a.m. - Talk 2: James Watson, University of Colorado, Boulder
  • 10:55 a.m. - Talk 3: Adam Norton, UMass Lowell
  • 11:15 a.m. - Break
  • 11:30 a.m. - Submitted talks
  • 12:30 p.m. - Break
  • 12:45 p.m. - Talk 4: Juan Arpacio, Siemens
  • 1:05 p.m. - Talk 5: Cindy Grimm, Oregon State University
  • 1:25 p.m. - Discussion
  • 2 p.m. - End


The activities of this workshop are funded by the National Institute of Standards and Technology (NIST) under award 70ANB19H101.