Toward Never-Ending Object Learning for Robots

Thesis (Ph.D.)--University of Washington, 2016-06

A household robot usually works in a complex working environment, where it will continuously see new objects and encounter new concepts in its lifetime. Therefore, being able to learn more objects is crucial for the robot to be continuously useful over its lifespan. Moving beyond previous object learning research problem, of which mostly focuses on learning with given training objects and concepts, this research addresses the problem of enabling a robot to learn new objects and concepts continuously. Specifically, our contributions are as follows: First, we study how to accurately identify target objects in scenes based on human users' language descriptions. We propose a novel identification system using an object's visual attributes and names to recognize objects. We also propose a method to enable the system to recognize objects based on new names without seeing any training instances of the names. The \textit{attribute-based identification system} improves both usability and accuracy over the previous ID-based object identification methods. Next, we consider the problem of organizing a large number of concepts into a semantic hierarchy. We propose a principle approach for creating semantic hierarchies of concepts via crowdsourcing. The approach can build hierarchies for various tasks and capture the uncertainty that naturally exists in these hierarchies. Experiments demonstrate that our method is more efficient, scalable, and accurate than previous methods. We also design a crowdsourcing evaluation to compare the hierarchies built by our method to expertly built ones. Results of the evaluation demonstrate that our approach outputs task-dependent hierarchies that can significantly improve user's performance of desired tasks. Finally, we build the first never-ending object learning framework, NEOL, that lets robots learn objects continuously. \neol\ automatically learns to organize object names into a semantic hierarchy using the crowdsourcing method we propose. It then uses the hierarchy to improve the consistency and efficiency of annotating objects. Further, it adapts information from additional image datasets to learn object classifiers from a very small number of training examples. Experiments show that NEOL significantly improves robots' accuracy and efficiency in learning objects over previous methods.