Improving robot vision models for object detection through interaction

We propose a method for learning specific object representations that can be applied (and reused) in visual detection and identification tasks. A machine learning technique called Cartesian Genetic Programming (CGP) is used to create these models based on a series of images. Our research investigates how manipulation actions might allow for the development of better visual models and therefore better robot vision. This paper describes how visual object representations can be learned and improved by performing object manipulation actions, such as, poke, push and pick-up with a humanoid robot. The improvement can be measured and allows for the robot to select and perform the `right' action, i.e. the action with the best possible improvement of the detector.

From vision to actions: Towards adaptive and autonomous humanoid robots

Although robotics research has seen advances over the last decades robots are still not in widespread use outside industrial applications. Yet a range of proposed scenarios have robots working together, helping and coexisting with humans in daily life. In all these a clear need to deal with a more unstructured, changing environment arises. I herein present a system that aims to overcome the limitations of highly complex robotic systems, in terms of autonomy and adaptation. The main focus of research is to investigate the use of visual feedback for improving reaching and grasping capabilities of complex robots. To facilitate this a combined integration of computer vision and machine learning techniques is employed. From a robot vision point of view the combination of domain knowledge from both imaging processing and machine learning techniques, can expand the capabilities of robots. I present a novel framework called Cartesian Genetic Programming for Image Processing (CGP-IP). CGP-IP can be trained to detect objects in the incoming camera streams and successfully demonstrated on many different problem domains. The approach requires only a few training images (it was tested with 5 to 10 images per experiment) is fast, scalable and robust yet requires very small training sets. Additionally, it can generate human readable programs that can be further customized and tuned. While CGP-IP is a supervised-learning technique, I show an integration on the iCub, that allows for the autonomous learning of object detection and identification. Finally this dissertation includes two proof-of-concepts that integrate the motion and action sides. First, reactive reaching and grasping is shown. It allows the robot to avoid obstacles detected in the visual stream, while reaching for the intended target object. Furthermore the integration enables us to use the robot in non-static environments, i.e. the reaching is adapted on-the- fly from the visual feedback received, e.g. when an obstacle is moved into the trajectory. The second integration highlights the capabilities of these frameworks, by improving the visual detection by performing object manipulation actions.

Reactive reaching and grasping on a humanoid: Towards closing the action-perception loop on the iCub

We propose a system incorporating a tight integration between computer vision and robot control modules on a complex, high-DOF humanoid robot. Its functionality is showcased by having our iCub humanoid robot pick-up objects from a table in front of it. An important feature is that the system can avoid obstacles - other objects detected in the visual stream - while reaching for the intended target object. Our integration also allows for non-static environments, i.e. the reaching is adapted on-the-fly from the visual feedback received, e.g. when an obstacle is moved into the trajectory. Furthermore we show that this system can be used both in autonomous and tele-operation scenarios.

Multi-edge decimation in multi-modal 3D collaborative applications

Multi-resolution modelling has become essential as modern 3D applications demand 3D objects with higher LODs (LOD). Multi-modal devices such as PDAs and UMPCs do not have sufficient resources to handle the original 3D objects. The increased usage of collaborative applications has created many challenges for remote manipulation working with 3D objects of different quality. This paper studies how we can improve multi-resolution techniques by performing multiedge decimation and using annotative commands. It also investigates how devices with poorer quality 3D object can participate in collaborative actions.

Type extension and efficient AST manipulation

Oberon-2 is an object-oriented language with a class structure based on type extension. The runtime structure of Oberon-2 is described and the low-level mechanism for dynamic type checking explained. It is shown that the superior type-safety of the language, when used for programming styles based on heterogeneous, pointer-linked data structures, has an entirely negligible cost in runtime performance.

Towards automatic object segmentation with sequential multiple views

Object segmentation is one of the fundamental steps for a number of robotic applications such as manipulation, object detection, and obstacle avoidance. This paper proposes a visual method for incorporating colour and depth information from sequential multiview stereo images to segment objects of interest from complex and cluttered environments. Rather than segmenting objects using information from a single frame in the sequence, we incorporate information from neighbouring views to increase the reliability of the information and improve the overall segmentation result. Specifically, dense depth information of a scene is computed using multiple view stereo. Depths from neighbouring views are reprojected into the reference frame to be segmented compensating for imperfect depth computations for individual frames. The multiple depth layers are then combined with color information from the reference frame to create a Markov random field to model the segmentation problem. Finally, graphcut optimisation is employed to infer pixels belonging to the object to be segmented. The segmentation accuracy is evaluated over images from an outdoor video sequence demonstrating the viability for automatic object segmentation for mobile robots using monocular cameras as a primary sensor.

Integrated health prediction of bridge systems using dynamic object oriented Bayesian networks (DOOBNS)

The serviceability and safety of bridges are crucial to people’s daily lives and to the national economy. Every effort should be taken to make sure that bridges function safely and properly as any damage or fault during the service life can lead to transport paralysis, catastrophic loss of property or even casualties. Nonetheless, aggressive environmental conditions, ever-increasing and changing traffic loads and aging can all contribute to bridge deterioration. With often constrained budget, it is of significance to identify bridges and bridge elements that should be given higher priority for maintenance, rehabilitation or replacement, and to select optimal strategy. Bridge health prediction is an essential underpinning science to bridge maintenance optimization, since the effectiveness of optimal maintenance decision is largely dependent on the forecasting accuracy of bridge health performance. The current approaches for bridge health prediction can be categorised into two groups: condition ratings based and structural reliability based. A comprehensive literature review has revealed the following limitations of the current modelling approaches: (1) it is not evident in literature to date that any integrated approaches exist for modelling both serviceability and safety aspects so that both performance criteria can be evaluated coherently; (2) complex system modelling approaches have not been successfully applied to bridge deterioration modelling though a bridge is a complex system composed of many inter-related bridge elements; (3) multiple bridge deterioration factors, such as deterioration dependencies among different bridge elements, observed information, maintenance actions and environmental effects have not been considered jointly; (4) the existing approaches are lacking in Bayesian updating ability to incorporate a variety of event information; (5) the assumption of series and/or parallel relationship for bridge level reliability is always held in all structural reliability estimation of bridge systems. To address the deficiencies listed above, this research proposes three novel models based on the Dynamic Object Oriented Bayesian Networks (DOOBNs) approach. Model I aims to address bridge deterioration in serviceability using condition ratings as the health index. The bridge deterioration is represented in a hierarchical relationship, in accordance with the physical structure, so that the contribution of each bridge element to bridge deterioration can be tracked. A discrete-time Markov process is employed to model deterioration of bridge elements over time. In Model II, bridge deterioration in terms of safety is addressed. The structural reliability of bridge systems is estimated from bridge elements to the entire bridge. By means of conditional probability tables (CPTs), not only series-parallel relationship but also complex probabilistic relationship in bridge systems can be effectively modelled. The structural reliability of each bridge element is evaluated from its limit state functions, considering the probability distributions of resistance and applied load. Both Models I and II are designed in three steps: modelling consideration, DOOBN development and parameters estimation. Model III integrates Models I and II to address bridge health performance in both serviceability and safety aspects jointly. The modelling of bridge ratings is modified so that every basic modelling unit denotes one physical bridge element. According to the specific materials used, the integration of condition ratings and structural reliability is implemented through critical failure modes. Three case studies have been conducted to validate the proposed models, respectively. Carefully selected data and knowledge from bridge experts, the National Bridge Inventory (NBI) and existing literature were utilised for model validation. In addition, event information was generated using simulation to demonstrate the Bayesian updating ability of the proposed models. The prediction results of condition ratings and structural reliability were presented and interpreted for basic bridge elements and the whole bridge system. The results obtained from Model II were compared with the ones obtained from traditional structural reliability methods. Overall, the prediction results demonstrate the feasibility of the proposed modelling approach for bridge health prediction and underpin the assertion that the three models can be used separately or integrated and are more effective than the current bridge deterioration modelling approaches. The primary contribution of this work is to enhance the knowledge in the field of bridge health prediction, where more comprehensive health performance in both serviceability and safety aspects are addressed jointly. The proposed models, characterised by probabilistic representation of bridge deterioration in hierarchical ways, demonstrated the effectiveness and pledge of DOOBNs approach to bridge health management. Additionally, the proposed models have significant potential for bridge maintenance optimization. Working together with advanced monitoring and inspection techniques, and a comprehensive bridge inventory, the proposed models can be used by bridge practitioners to achieve increased serviceability and safety as well as maintenance cost effectiveness.

Metastability and emergent performance of dynamic interceptive actions

Objectives: Adaptive patterning of human movement is context specific and dependent on interacting constraints of the performer–environment relationship. Flexibility of skilled behaviour is predicated on the capacity of performers to move between different states of movement organisation to satisfy dynamic task constraints, previously demonstrated in studies of visual perception, bimanual coordination, and an interceptive combat task. Metastability is a movement system property that helps performers to remain in a state of relative coordination with their performance environments, poised between multiple co-existing states (stable and distinct movement patterns or responses). The aim of this study was to examine whether metastability could be exploited in externally paced interceptive actions in fast ball sports, such as cricket. Design: Here we report data on metastability in performance of multi-articular hitting actions by skilled junior cricket batters (n = 5). Methods: Participants’ batting actions (key movement timings and performance outcomes) were analysed in four distinct performance regions varied by ball pitching (bounce) location. Results: Results demonstrated that, at a pre-determined distance to the ball, participants were forced into a meta-stable region of performance where rich and varied patterns of functional movement behaviours emerged. Participants adapted the organisation of responses, resulting in higher levels of variability in movement timing in this performance region, without detrimental effects on the quality of interceptive performance outcomes. Conclusions: Findings provide evidence for the emergence of metastability in a dynamic interceptive action in cricket batting. Flexibility and diversity of movement responses were optimised using experiential knowledge and careful manipulation of key task constraints of the specific sport context.

Semantic context and visual feature effects in object naming : an fMRI study using arterial spin labeling

Previous behavioral studies reported a robust effect of increased naming latencies when objects to be named were blocked within semantic category, compared to items blocked between category. This semantic context effect has been attributed to various mechanisms including inhibition or excitation of lexico-semantic representations and incremental learning of associations between semantic features and names, and is hypothesized to increase demands on verbal self-monitoring during speech production. Objects within categories also share many visual structural features, introducing a potential confound when interpreting the level at which the context effect might occur. Consistent with previous findings, we report a significant increase in response latencies when naming categorically related objects within blocks, an effect associated with increased perfusion fMRI signal bilaterally in the hippocampus and in the left middle to posterior superior temporal cortex. No perfusion changes were observed in the middle section of the left middle temporal cortex, a region associated with retrieval of lexical-semantic information in previous object naming studies. Although a manipulation of visual feature similarity did not influence naming latencies, we observed perfusion increases in the perirhinal cortex for naming objects with similar visual features that interacted with the semantic context in which objects were named. These results provide support for the view that the semantic context effect in object naming occurs due to an incremental learning mechanism, and involves increased demands on verbal self-monitoring.

Experiences with an Object-Oriented, Multi-Stage Language

Metaphor is a multi-stage programming language extension to an imperative, object-oriented language in the style of C# or Java. This paper discusses some issues we faced when applying multi-stage language design concepts to an imperative base language and run-time environment. The issues range from dealing with pervasive references and open code to garbage collection and implementing cross-stage persistence.

Review article : ecological approach in studying the relation of perception-action : review of the theoretical and empirical data

In daily activities people are using a number of available means for the achievement of balance, such as the use of hands and the co-ordination of balance. One of the approaches that explains this relationship between perception and action is the ecological theory that is based on the work of a) Bernstein (1967), who imposed the problem of ‘the degrees of freedom’, b) Gibson (1979), who referred to the theory of perception and the way which the information is received from the environment in order for a certain movement to be achieved, c) Newell (1986), who proposed that movement can derive from the interaction of the constraints that imposed from the environment and the organism and d) Kugler, Kelso and Turvey (1982), who showed the way which “the degrees of freedom” are connected and interact. According to the above mentioned theories, the development of movement co-ordination can result from the different constraints that imposed into the organism-environment system. The close relation between the environmental and organismic constraints, as well as their interaction is responsible for the movement system that will be activated. These constraints apart from shaping the co-ordination of specific movements can be a rate limiting factor, to a certain degree, in the acquisition and mastering of a new skill. This frame of work can be an essential tool for the study of catching an object (e.g., a ball). The importance of this study becomes obvious due to the fact that movements that involved in catching an object are representative of every day actions and characteristic of the interaction between perception and action.

Why, how, what for? : motivations, actions and expectations in habitual entrepreneurship

Principal topic: Is habitual entrepreneurship different? Answering this is important to the field, however there is little systematic evidence, thus far. We addresses this by examining the role experience plays at three possible points of difference: motivations, actions and expectations; and by comparing those currently in the process of starting a business with those who have recent success in business creation. Firstly, we assess the balance of opportunity versus necessity motivation, internally versus externally stimulated decision processes and future growth aspirations. Literature suggests novices are more likely motivated to nascency out of necessity, and favour a manageable business size, while habitual entrepreneurs are more likely motivated by internally stimulated or idea driven processes. Secondly, we examine actions undertaken by successful experienced founders during gestation, contrasting ‘information collection’ and ‘opportunity definition’. Drawing on prior research we expect novices more likely to have enacted ‘information search’ while habitual entrepreneurs enact ‘opportunity definition’. Thirdly, we examine perceptions of venture success, where findings on overconfidence suggest that habitual entrepreneurs expect a higher chance of success for their ventures, while inexperience leads novices to underestimate the difficulty of entrepreneurial survival. Method: Empirical evidence to test these conjectures was drawn from a screened random sample of over 1100 Australian nascent and newly started business ventures. This information was collected during 2007/8 using a telephone survey. Results and Implications: Why do habitual entrepreneurs keep coming back? Findings suggest that while the pursuit of opportunity is shared by novice and experienced entrepreneur alike, consideration of repeat entrepreneurship may be motivated by a desire for growth. While idea driven motivations might not delineate a distinction during nascency, it does seem to be a factor contributing to the success of young firms. This warrants further research. How do habitual entrepreneurs behave differently? It seems they act to clearly define market opportunities as a matter of priority during venture gestation. What effect does entrepreneurial experience have on future expectations? Clearly a sense of realism is drawn over the difficulties that might be faced, and accords more circumspect judgements of venture survival. This finding informs practitioners considering entrepreneurship for the first time.

Where the actions are : bring reality into virtual reality

This paper describes a work-in-progress on developing design environments that combine wireless and mobile technologies with augmented reality to facilitate bringing context from the physical environment to the virtual models for design work. One of the challenges for designers in a variety of end-user-oriented design disciplines such as architecture and industrial design has been capturing and replaying the contextual information of the intended domain of the artifact being designed. Either the technology is decidedly low-tech, such as charcoal drawings in a sketchbook, out-of-reach, such as immersive virtual reality CAVEs, or a “make-do” with existing technologies, such as a collage of digital photos. This paper describes a novel combination of “off-the-shelf” technologies that may allow designers more capability to create models using standard computer-aided design applications and augmented reality to combine the current, physical context with the projected, digital context. We demonstrate this approach in the building design domain to address a common problem in building construction, construction defect resolution.