Spatiotemporal coherent semantic maps with mobile robots

Abstract

Along the last decades persistent research endeavors in the areas of robotics andartificial intelligence revealed the great challenge of robot navigation, an areathat combines robot mobility with perception of the environment. Moreover,in modern human societies it is of great importance to build up machines thatcan be operated by non specialists or even by technologically illiterate people,such as youngsters or elderly. Therefore, the mobile robots to be released intothe market in the near future should possess, among others, the potential ofproducing meaningful internal perceptual representations of their own environment,capacitating them to cope a range of real-life situations and tasks. Tomake matters even more challenging, when it comes for mapping and navigation,robots should comprehend human concepts about places and objects,to skillfully deploy in human frequented environments. In response to thischallenge, intense research efforts, to build cognitive robots apt to competentlyperceive and understand their surroundings and to cooperate with humans,take place. With this goal in view, semantic mapping with mobile robots can constituteto a holistic solution in response to the aforementioned challenges. Thesemantic mapping is an augmented representation of the robot’s environmentthat -supplementary to the geometrical knowledge- encapsulates characteristicscompatible with human understanding. It provides several algorithmic opportunitiesfor innovative development of applications that will eventually lead tothe human robot interaction. The main objective of the PhD dissertation in handis the construction of accurate and consistent semantic maps facilitating amplerobot deployment in domestic environments.The motivation behind this PhD dissertation has been the observation thatthe plethora of the existing mapping and navigation algorithms are not ableto provide a sufficient representation of the environment in terms of humans’concepts. This is due to the fact that the mapping methods developed sofar focus on the construction of geometrical maps. Although some of thesesolutions proved to be capable of driving robots into specific target positions,they lack of high level cognition attributes, which would allow them to bring the human-robot interaction one step beyond. Aiming to remove this barrier, thisdissertation is oriented towards the direction of developing semantic mappingalgorithms for high level robot navigation. Therefore, this doctoral researchidentified the basic components of the semantic mapping and developed aninnovative solution for each one of them. Due to the fact that the semanticmapping requires an integrated system that comprises several subordinatemodules, a wide range of a algorithms that serve different tasks had to bedesigned and implemented. Within the context of this thesis several algorithmicmodules have been developed including a competent localization algorithm,novel simultaneously localization and mapping strategies, breakthrough place andobject recognition tactics, as well as the integration of all these methods undera time supervised framework able to produce consistent semantic maps. Dueto the fact that each subordinate module comprises an innovative solutionto the respective field, the resulting semantic mapping system constitutesa state of the art solution in the area of conceptual mapping with mobilerobots. The introduced algorithms exploit solely visual and depth sensors,while by combining basic tools from three different scientific areas such asrobotics, computer vision and machine learning the final objective is accomplished.Overall, the main contribution of this thesis to the advancement the stateof the art is the introduction of a stacked map hierarchy of four differenttype of maps, namely a metric, a topological, a labeled sparse topological andan augmented navigation one. Each of these representations accomplishes aunique purpose: (i) the metric is the physical (lower) layer; (ii) the topologicalone contains abstract geometrical information of the environment, i.e. pointclouds registered in a graph of nodes; (iii) the labeled sparse topological oneestablishes the spatiotemporal coherence by associating the respective nodes inthe topometric maps via place labels and geometrical transformations, enablingbidirectional exchange of information among the conceptual and metric mapsand, last, (iv) the augmented navigation map inheres the significance of thedetected places as well as their connectivity relationships, expressed in termsof their transition probability.The thesis in hand has been developed in a hierarchical fashion and can bedivided into four main chapters. The first one comprises a literature survey ofthe existing semantic mapping methods in which an explicit analysis of the sofar developed methods is sought. The insights of the semantic mapping arereviewed, the distinct components encompassing, to give a categorization of therelated literature, are studied the possible applications in mobile robotics areexamined and, lastly, the methods and databases available for benchmarkingare referred. Furthermore, a quality-based taxonomy of the existing semanticmapping methods highlights the dominant attributes such methods retain.More precisely, according to the scale, to which each method is expanded, the metric map could be either a single scene or a progressively created map.Another important attribute a typical semantic mapping method possessesis the existence of the respective topological map, that is an abstraction ofthe explored environment in terms of a graph. The nodes of such a graph areorganized in a geometrical manner, so as to simultaneously preserve conceptualknowledge about the explored places. Moreover, the modalities (single ormultiple visual cues) utilized to reason about the observed scene constitute anelement apt to distinguish the abundance of different methods. An additionalfeature in many recent semantic mapping techniques is the temporal coherencesuch a map reveals, which renders it useful for high-level activities, viz. taskplanning or human robot interaction.The second chapter refers to the description of the developed technologicalbackground required to build a consistent semantic map. Therefore, a majorcontribution of the first part is the development of an innovative visual odometryalgorithm able to operate in real time. This algorithm receives as input successivestereo pair of images from a stereoscopic camera mounted on a mobilerobot. It involves the detection of the salient landmarks between successiveimages. A depth estimation of these features is then obtained and a novelnon-iterative outlier detection and discarding methodology able to remove boththe mismatches between the features and the inserted errors due to the 3Dreconstruction procedure. A hierarchical motion estimation technique, whichproduces robust estimations for the movement of the robot is then adopted,thus providing refinements to the robot’s global position and orientation. Anadditional 3D reconstruction algorithm that operates on stereo images hasbeen developed providing accurate reconstruction of the area observed bythe robot. Moreover, the localization algorithm comprises the cornerstone forthe development of a simultaneously localization and mapping system suitablefor the 3D geometrical mapping of the explored environment. This 3D metricmapping system is based solely on an RGB-D sensor, where in course of robot’slocomotion 3D point clouds are merged with respect to the visual odometry.The resulted 3D map is refined by exploiting a random sample consensus planedetection algorithm accompanied by an iterative closest point registration step,among the dominant planes of the consecutive time instances, resulting thusin a very consistent geometrical 3D map. All the aforementioned developedalgorithms have been evaluated on a custom made robot platform bearing twostereoscopic cameras with different baselines and a RGB-D sensor.The third chapter encloses all the semantic mapping methods based onvisual cues that have been developed within this PhD dissertation. The firstone examines the overall traversability of the observed scene taking into considerationthe robot’s embodiment. This knowledge constitutes a cornerstone forthe autonomous robot navigation. The developed system utilizes an algorithm to retrieve specific characteristics of the environment using a stereo cameraand to produce a disparity map of the scene. Then, the v-disparity image iscalculated based on the disparity map. The v-disparity is then exploited by afeature extraction procedure to provide the system with respective conceptualvectors, which are used to train support vector machines in order to assess theoverall traversability of the scene. The traversable classified scenes are furtherprocessed and the likelihood distribution of the collision risk assessment, whenthe robot moves towards any direction, is calculated. The second part involvesthe description of a novel object recognition algorithm based on hierarchical temporalmemory networks. This constitutes a supervised learning method usedto recognize objects in different orientations. It introduces specific alternativerules for the design of each building block of a hierarchical temporal memorynetwork. These rules expand both the spatial and the temporal module ofthe network. Various type of input layers have been tested such as logppolarand saliency detection ones in order to find the solution that fits better in applicationsthat concern cluttered environments. The third part of this chaptercomprises the description of an innovative place classification algorithm. Withinthis work, in course of robot’s locomotion, salient visual features are detectedand they shape a bag-of-features problem, quantized by a neural gas to codethe spatial information for each scene. Each input image is transformed intoan appearance based histogram representation that abstracts the place in avery compatible and consistent manner. The learning procedure is performedby support vector machines able to accurately recognize multiple dissimilarplaces. The innovative appearance based algorithm produces semantic inferencessuitable for labeling unexplored environments. In the rest of this chaptera long range semantic mapping framework is described, where geometricalmapping, with place and object recognition is combined in order to construct ahuman oriented semasiological map. This framework features geometrical andsemasiological attributes capable to reveal relationships between objects andplaces in a real-life environment. The geometrical component consists of a 3Dmap, onto which a topological map is deployed, representing the explored areaas a graph of nodes. The semasiological part is realized by putting together aplace recognition algorithm and an object recognition one. The categorizationof the different places relies on the resolution of appearance-based consistencyhistograms, while for the recognition of objects in the scene we make use of thehierarchical temporal memory network boosted by a saliency attentional model.These semantical attributes are then deposited on each node of the topologicalmap, in order to augment it with the belief distributions regarding the visitedplaces, thus resulting in a 3D object map embodying geometrical and semanticalproperties.Finally, in the fourth chapter of this thesis a novel semantic mapping method suitable for robot navigation in a human compatible manner is illustrated. Themain objective of this chapter is to take into account the time proximity of therobot acquired frames within the semantic map. The goal of this method is; (i)to introduce a semasiological mapping method for robot exploration and (ii) tomake use of the constructed semantic map as a means to provide a hierarchicalnavigation solution. The semantic map is formed during the robot’s course,relying on the memorization of abstract place representations. It integrates thespace quantization, the time proximity of the acquired frames and the spatialcoherence in terms of a novel labeled sparse topological map. A time evolvingaugmented navigation graph is shaped determining the semantic topology ofthe explored environment and the physical connectivity among the recognizedplaces expressed by the inter-place transition probability. Concerning therobot navigation part, a human-robot interaction methodology is illustratedcapable of competently addressing go-to commands. As a product of thismodule, an augmented navigation graph is formed, which handles the high levelrobot navigation, forwarding the presumable sequence of places the robotshould traverse to reach its target location. Accordingly, for the low levellocal navigation the topometric data of the semantic map are made use of.Additionally, to assist the human robot interaction a graphical user interfacehas been developed providing an overall supervision of the mapping andnavigation procedure.The last chapter of this dissertation concludes the doctoral research conductedhereby. It discusses the achievements of this work, while it also highlightsthe open issues and the questions revealed during the developmentof the several algorithmic solutions. Additionally, some future trends of thesemantic mapping are outlined establishing the potential descendants of thisdissertation.