RoboErgoSum Project

Situation awareness and semantic scene interpretation.

Scene understanding has traditionally been - and mostly still is - addressed as a process of observation. Even if active vision was introduced by [1], this was to gain more information through exploration of objects or viewpoint selection. We believe that treating perception as an observation process only is the very reason for which robotic perception methods are to date unable to provide a general capacity of scene understanding. On the other hand, work in neuroscience has shown that there is a strong relationship between perception and action. Considering perception in relation to action requires to interpret the scene in terms of the agent’s own potential activity on the one hand, which very naturally fits with the notion of Gibson’s affordances [2], but also - and that’s much less frequently discussed - in terms of other agents actions. We find in the mirror neurons of the premotor cortex [3] indications on the possibility of a biological foundation of this latter view.

Reasoning jointly on perception and action requires self-localization with respect to the environment. Hence developing visuo-motor representations and not just environment representations puts the robot "self" in the center of the perceptual process, and provides for a link between self-awareness and situation-awareness. Indeed, robot localization with respect to its environment is one expression of the differentiation between the robot’s body and the "external the world", and includes a necessary distinction between its parts and surrounding objects. There are several robot localization methods, mostly based on probabilistiic representations, and running concurrently within mapping processes in SLAM (the wide literature on the subject is summarized in [4]). Those methods rely on Bayesian filters, and we will use and develop such methods. However in the way they are exploited in the literature, they suppose the robot able to distinguish the environment from itself, and therefore completely bypass the problem of self-awareness. Reasoning on robot displacements together with the perceptual data on the changing and unchanging scene surrounding it, to find invariants and regularities is a primary source of self-awareness because some of them express robot shape. Classification processes will be used to detect those regularities.

Another aspect of the role of self-awareness in perception is semantic interpretation. The absence of semantic interpretation is a well-known shortcoming, and a yet unsolved difficulty in robot perception. We hypothesize that the proper way to address this problem is to relate scene interpretation to self-awareness, which means that understanding a scene has necessarily to be considered as a process involving a positioning of the observer in it or with respect to it, and extracting the observer’s relationships to scene components. Those relationships are a basis of scene understanding and for forming intelligible representations related to robot actions.

Back to Work Packages