Multi-scale cortical keypoints for realtime hand tracking and gesture recognition

Human-robot interaction is an interdisciplinary research area which aims at integrating human factors, cognitive psychology and robot technology. The ultimate goal is the development of social robots. These robots are expected to work in human environments, and to understand behavior of persons through gestures and body movements. In this paper we present a biological and realtime framework for detecting and tracking hands. This framework is based on keypoints extracted from cortical V1 end-stopped cells. Detected keypoints and the cells’ responses are used to classify the junction type. By combining annotated keypoints in a hierarchical, multi-scale tree structure, moving and deformable hands can be segregated, their movements can be obtained, and they can be tracked over time. By using hand templates with keypoints at only two scales, a hand’s gestures can be recognized.

BIMP: A real-time biological model of multi-scale keypoint detection in V1

We present an improved, biologically inspired and multiscale keypoint operator. Models of single- and double-stopped hypercomplex cells in area V1 of the mammalian visual cortex are used to detect stable points of high complexity at multiple scales. Keypoints represent line and edge crossings, junctions and terminations at fine scales, and blobs at coarse scales. They are detected by applying first and second derivatives to responses of complex cells in combination with two inhibition schemes to suppress responses along lines and edges. A number of optimisations make our new algorithm much faster than previous biologically inspired models, achieving real-time performance on modern GPUs and competitive speeds on CPUs. In this paper we show that the keypoints exhibit state-of-the-art repeatability in standardised benchmarks, often yielding best-in-class performance. This makes them interesting both in biological models and as a useful detector in practice. We also show that keypoints can be used as a data selection step, significantly reducing the complexity in state-of-the-art object categorisation. (C) 2014 Elsevier B.V. All rights reserved.

Painterly rendering using human vision

Painterly rendering has been linked to computer vision, but we propose to link it to human vision because perception and painting are two processes that are interwoven. Recent progress in developing computational models allows to establish this link. We show that completely automatic rendering can be obtained by applying four image representations in the visual system: (1) colour constancy can be used to correct colours, (2) coarse background brightness in combination with colour coding in cytochrome-oxidase blobs can be used to create a background with a big brush, (3) the multi-scale line and edge representation provides a very natural way to render fi ner brush strokes, and (4) the multi-scale keypoint representation serves to create saliency maps for Focus-of-Attention, and FoA can be used to render important structures. Basic processes are described, renderings are shown, and important ideas for future research are discussed.

Improved line/edge detection and visual reconstruction

Lines and edges provide important information for object categorization and recognition. In addition, one brightness model is based on a symbolic interpretation of the cortical multi-scale line/edge representation. In this paper we present an improved scheme for line/edge extraction from simple and complex cells and we illustrate the multi-scale representation. This representation can be used for visual reconstruction, but also for nonphotorealistic rendering. Together with keypoints and a new model of disparity estimation, a 3D wireframe representation of e.g. faces can be obtained in the future.

Looking through the eyes of the painter: from visual perception to non-photorealistic rendering

In this paper we present a brief overview of the processing in the primary visual cortex, the multi-scale line/edge and keypoint representations, and a model of brightness perception. This model, which is being extended from 1D to 2D, is based on a symbolic line and edge interpretation: lines are represented by scaled Gaussians and edges by scaled, Gaussian-windowed error functions. We show that this model, in combination with standard techniques from graphics, provides a very fertile basis for non-photorealistic image rendering.

Artistic rendering of the visual cortex

In this paper we explain the processing in the first layers of the visual cortex by simple, complex and endstopped cells, plus grouping cells for line, edge, keypoint and saliency detection. Three visualisations are presented: (a) an integrated scheme that shows activities of simple, complex and end-stopped cells, (b) artistic combinations of selected activity maps that give an impression of global image structure and/or local detail, and (c) NPR on the basis of a 2D brightness model. The cortical image representations offer many possibilities for non-photorealistic rendering.

Correlates of facial expressions in the primary visual cortex

Face detection and recognition should be complemented by recognition of facial expression, for example for social robots which must react to human emotions. Our framework is based on two multi-scale representations in cortical area V1: keypoints at eyes, nose and mouth are grouped for face detection [1]; lines and edges provide information for face recognition [2].

A vision system for detecting paths and moving obstacles for the blind

In this paper we present a monocular vision system for a navigation aid. The system assists blind persons in following paths and sidewalks, and it alerts the user to moving obstacles which may be on collision course. Path borders and the vanishing point are de-tected by edges and an adapted Hough transform. Opti-cal flow is detected by using a hierarchical, multi-scale tree structure with annotated keypoints. The tree struc-ture also allows to segregate moving objects, indicating where on the path the objects are. Moreover, the centre of the object relative to the vanishing point indicates whether an object is approaching or not.

Disparity energy model with keypoint disparity validation

A biological disparity energy model can estimate local depth information by using a population of V1 complex cells. Instead of applying an analytical model which explicitly involves cell parameters like spatial frequency, orientation, binocular phase and position difference, we developed a model which only involves the cells’ responses, such that disparity can be extracted from a population code, using only a set of previously trained cells with random-dot stereograms of uniform disparity. Despite good results in smooth regions, the model needs complementary processing, notably at depth transitions. We therefore introduce a new model to extract disparity at keypoints such as edge junctions, line endings and points with large curvature. Responses of end-stopped cells serve to detect keypoints, and those of simple cells are used to detect orientations of their underlying line and edge structures. Annotated keypoints are then used in the leftright matching process, with a hierarchical, multi-scale tree structure and a saliency map to segregate disparity. By combining both models we can (re)define depth transitions and regions where the disparity energy model is less accurate.

Object segregation and local gist vision using low-level geometry

Multi-scale representations of lines, edges and keypoints on the basis of simple, complex and end-stopped cells can be used for object categorisation and recognition (Rodrigues and du Buf, 2009 BioSystems 95 206-226). These representations are complemented by saliency maps of colour, texture, disparity and motion information, which also serve to model extremely fast gist vision in parallel with object segregation. We present a low-level geometry model based on a single type of self-adjusting grouping cell, with a circular array of dendrites connected to edge cells located at several angles.

Discards from experimental trammel nets in southern European small-scale fisheries

Trammel net discards in four southern European areas were considerable, with a total of 137 species (79.7% of the total) discarded 65, 105, 46 and 32 species in the Basque country (Spain), Algarve (Portugal), Gulf of Cadiz (Spain) and Cyclades islands (Greece), respectively. The overall discard rate in terms of catch numbers ranged from 15% for the Cyclades to 49% for the Algarve, with the high discard rate for the latter due largely to small pelagic fishes. Discards in the four areas consisted mainly of Trisopterus luscus (Basque country), Scomber japonicus (Algarve), Torpedo torpedo (Cadiz) and Sardina pilchardus (all three areas), and Diplodus annularis in the Cyclades. Strong seasonal variation in discarding was found, reflecting differences in metiers and the versatility of trammel nets as a gear. Discarding, both in terms of numbers of species and individuals decreased with increasing inner panel mesh size. The main reasons for discarding were: (1) species of no or low commercial value (e.g. Scomber japonicus; Torpedo torpedo), (2) commercial species that were damaged or spoiled (e.g. Merluccius merluccius), (3) undersized commercial species (e.g. Lophius piscatorius), and (4) species of commercial value but not caught in sufficient quantities to warrant sale (e.g. Sardina pilchardus). A decrease in soak time together with the appropriate choice of mesh sizes could contribute to a reduction in discarding and to improved sustainability and use of scarce resources in the small-scale, inshore multi-species fisheries of southern Europe. (C) 2007 Elsevier B.V. All rights reserved.

Quantifying the roles of competing static gears: comparative selectivity of longlines and monofilament gill nets in a multi-species fishery of the Algarve (southern Portugal)

Fishing trials with monofilament gill nets and longlines using small hooks were carried out in Algarve waters (southern Portugal) over a one-year period. Four hook sizes of "Mustad" brand, round bent, flatted sea hooks (Quality 2316 DT, numbers 15, 13, 12 and 11) and four mesh sizes of 25, 30, 35 and 40 mm (bar length) monofilament gill nets were used. Commercially valuable sea breams dominated the longline catches while small pelagics were relatively more important in the gill nets. Significant differences in the catch size frequency distributions of the two gears were found for all the most important species caught by both gears (Boops boops, Diplodus bellottii, Diplodus vulgaris, Pagellus acarne, Pagellus erythrinus, Spondyiosoma cantharus, Scomber japonicus and Scorpaena notata), with longlines catching larger fish and a wider size range than nets. Whereas longline catch size frequency distributions for most species for the different hook sizes were generally highly overlapped, suggesting little or no differences in size selectivity, gill net catch size frequency distributions clearly showed size selection. A variety of models were fitted to the gill net and hook data using the SELECT method, while the parameters of the logistic model were estimated by maximum likelihood for the longline data. The bi-normal model gave the best fits for most of the species caught with gill nets, while the logistic model adequately described hook selectivity. The results of this study show that the two static gears compete for many of the same species and have different impacts in terms of catch composition and size selectivity. This information will I;e useful for the improved management of these small-scale fisheries in which many different gears compete for scarce resources.

A multi-model approach to evaluate the role of environmental variability and fishing pressure in sardine fisheries

Understanding the fluctuations in population abundance is a central question in fisheries. Sardine fisheries is of great importance to Portugal and is data-rich and of primary concern to fisheries managers. In Portugal, sub-stocks of Sardina pilchardus (sardine) are found in different regions: the Northwest (IXaCN), Southwest (IXaCS) and the South coast (IXaS-Algarve). Each of these sardine sub-stocks is affected differently by a unique set of climate and ocean conditions, mainly during larval development and recruitment, which will consequently affect sardine fisheries in the short term. Taking this hypothesis into consideration we examined the effects of hydrographic (river discharge), sea surface temperature, wind driven phenomena, upwelling, climatic (North Atlantic Oscillation) and fisheries variables (fishing effort) on S. pilchardus catch rates (landings per unit effort, LPUE, as a proxy for sardine biomass). A 20-year time series (1989-2009) was used, for the different subdivisions of the Portuguese coast (sardine sub-stocks). For the purpose of this analysis a multi-model approach was used, applying different time series models for data fitting (Dynamic Factor Analysis, Generalised Least Squares), forecasting (Autoregressive Integrated Moving Average), as well as Surplus Production stock assessment models. The different models were evaluated, compared and the most important variables explaining changes in LPUE were identified. The type of relationship between catch rates of sardine and environmental variables varied across regional scales due to region-specific recruitment responses. Seasonality plays an important role in sardine variability within the three study regions. In IXaCN autumn (season with minimum spawning activity, larvae and egg concentrations) SST, northerly wind and wind magnitude were negatively related with LPUE. In IXaCS none of the explanatory variables tested was clearly related with LPUE. In IXaS-Algarve (South Portugal) both spring (period when large abundances of larvae are found) northerly wind and wind magnitude were negatively related with LPUE, revealing that environmental effects match with the regional peak in spawning time. Overall, results suggest that management of small, short-lived pelagic species, such as sardine quotas/sustainable yields, should be adapted to a regional scale because of regional environmental variability.

Broad-scale mapping of seafloor habitats in the north-east Atlantic using existing environmental data

If marine management policies and actions are to achieve long-term sustainable use and management of the marine environment and its resources, they need to be informed by data giving the spatial distribution of seafloor habitats over large areas. Broad-scale seafloor habitat mapping is an approachwhich has the benefit of producing maps covering large extents at a reasonable cost. This approach was first investigated by Roff et al. (2003), who, acknowledging that benthic communities are strongly influenced by the physical characteristics of the seafloor, proposed overlaying mapped physical variables using a geographic information system (GIS) to produce an integrated map of the physical characteristics of the seafloor. In Europe the method was adapted to the marine section of the EUNIS (European Nature Information System) classification of habitat types under the MESH project, andwas applied at an operational level in 2011 under the EUSeaMap project. The present study compiled GIS layers for fundamental physical parameters in the northeast Atlantic, including (i) bathymetry, (ii) substrate type, (iii) light penetration depth and (iv) exposure to near-seafloor currents andwave action. Based on analyses of biological occurrences, significant thresholds were fine-tuned for each of the abiotic layers and later used in multi-criteria raster algebra for the integration of the layers into a seafloor habitat map. The final result was a harmonised broad-scale seafloor habitat map with a 250 m pixel size covering four extensive areas, i.e. Ireland, the Bay of Biscay, the Iberian Peninsula and the Azores. The map provided the first comprehensive perception of habitat spatial distribution for the Iberian Peninsula and the Azores, and fed into the initiative for a pan- European map initiated by the EUSeaMap project for Baltic, North, Celtic and Mediterranean seas.