Looking through the eyes of the painter: from visual perception to non-photorealistic rendering

In this paper we present a brief overview of the processing in the primary visual cortex, the multi-scale line/edge and keypoint representations, and a model of brightness perception. This model, which is being extended from 1D to 2D, is based on a symbolic line and edge interpretation: lines are represented by scaled Gaussians and edges by scaled, Gaussian-windowed error functions. We show that this model, in combination with standard techniques from graphics, provides a very fertile basis for non-photorealistic image rendering.

Brightness perception: one simple model explaines most if not all effects

Few models can explain Mach bands (Pessoa, 1996 Vision Research 36 3205-3227) . Our own employs multiscale line and edge coding by simple and complex cells. Lines are interpreted by Gaussian functions, edges by bipolar, Gaussian-truncated errorfunctions. Widths of these functions are coupled to the scales of the underlying cells and the amplitudes are determined by their responses.

Perception-based painterly rendering: funcionality and interface design

Painterly rendering (non-photorealistic rendering or NPR) aims at translating photographs into paintings with discrete brush strokes, simulating certain techniques (im- or expressionism) and media (oil or watercolour). Recently, our research into visual perception and models of processes in the visual cortex resulted in a new rendering scheme, in which detected lines and edges at different scales are translated into brush strokes of different sizes. In order to prepare a version which is suitable for many users, including children, the design of the interface in terms of window and menu system is very important. Discussions with artists and non-artists led to three design criteria: (1) the interface must reflect the procedures and possibilities that real painters follow and use, (2) it must be based on only one window, and (3) the menu system must be very simple, avoiding a jungle of menus and sub-menus. This paper explains the interface that has been developed.

Face segregation and recognition by cortical multi-scale line and edge coding

Models of visual perception are based on image representations in cortical area V1 and higher areas which contain many cell layers for feature extraction. Basic simple, complex and end-stopped cells provide input for line, edge and keypoint detection. In this paper we present an improved method for multi-scale line/edge detection based on simple and complex cells. We illustrate the line/edge representation for object reconstruction, and we present models for multi-scale face (object) segregation and recognition that can be embedded into feedforward dorsal and ventral data streams (the “what” and “where” subsystems) with feedback streams from higher areas for obtaining translation, rotation and scale invariance.

Face recognition by cortical multi-scale line and edge representations

Empirical studies concerning face recognition suggest that faces may be stored in memory by a few canonical representations. Models of visual perception are based on image representations in cortical area V1 and beyond, which contain many cell layers for feature extraction. Simple, complex and end-stopped cells provide input for line, edge and keypoint detection. Detected events provide a rich, multi-scale object representation, and this representation can be stored in memory in order to identify objects. In this paper, the above context is applied to face recognition. The multi-scale line/edge representation is explored in conjunction with keypoint-based saliency maps for Focus-of-Attention. Recognition rates of up to 96% were achieved by combining frontal and 3/4 views, and recognition was quite robust against partial occlusions.

An integrated framework for combining gist vision with object segregation categorisation and recognition

There are roughly two processing systems: (1) very fast gist vision of entire scenes, completely bottom-up and data driven, and (2) Focus-of-Attention (FoA) with sequential screening of specific image regions and objects. The latter system has to be sequential because unnormalised input objects must be matched against normalised templates of canonical object views stored in memory, which involves dynamic routing of features in the visual pathways.

Simple brightness models with low-pass and Gabor filters

A syntactical brightness model based on a multiscale line and edge representation obtained by a set of anisotropic Gabor filters is quite complex (du Buf and Fischer, 1995 Optical Engineering 34 1900-1911). Although only tested in 1-D, it was shown to yield correct brightness effects for many patterns.

Face normalization using multi-scale cortical keypoints

Empirical studies concerning face recognition suggest that faces may be stored in memory by a few canonical representations. Models of visual perception are based on image representations in cortical area V1 and beyond, which contain many cell layers for feature extractions. Simple, complex and end-stopped cells tuned to different spatial frequencies (scales) and/or orientations provide input for line, edge and keypoint detection. This yields a rich, multi-scale object representation that can be stored in memory in order to identify objects. The multi-scale, keypoint-based saliency maps for Focus-of-Attention can be explored to obtain face detection and normalization, after which face recognition can be achieved using the line/edge representation. In this paper, we focus only on face normalization, showing that multi-scale keypoints can be used to construct canonical representations of faces in memory.

Multi-scale keypoint hierarchy for Focus-of-Attention and object detection

Hypercolumns in area V1 contain frequency- and orientation-selective simple and complex cells for line (bar) and edge coding, plus end-stopped cells for key- point (vertex) detection. A single-scale (single-frequency) mathematical model of single and double end-stopped cells on the basis of Gabor filter responses was developed by Heitger et al. (1992 Vision Research 32 963-981). We developed an improved model by stabilising keypoint detection over neighbouring micro- scales.

Fine arts edutainment: the amateur painter

A new scheme for painterly rendering (NPR) has been developed. This scheme is based on visual perception, in particular themulti-scale line/edge representation in the visual cortex. The Amateur Painter (TAP) is the user interface on top of the rendering scheme. It allows to (semi)automatically create paintings from photographs, with different types of brush strokes and colour manipulations. In contrast to similar painting tools, TAP has a set of menus that reflects the procedure followed by a normal painter. In addition, menus and options have been designed such that they are very intuitive, avoiding a jungle of sub-menus with options from image processing that children and laymen do not understand. Our goal is to create a tool that is extremely easy to use, with the possibility that the user becomes interested in painting techniques, styles, and fine arts in general.

Recognition of facial expressions by cortical multi-scale line and edge coding

Empirical studies concerning face recognition suggest that faces may be stored in memory by a few canonical representations. Models of visual perception are based on image representations in cortical area V1 and beyond, which contain many cell layers for feature extraction. Simple, complex and end-stopped cells provide input for line, edge and keypoint detection. Detected events provide a rich, multi-scale object representation, and this representation can be stored in memory in order to identify objects. In this paper, the above context is applied to face recognition. The multi-scale line/edge representation is explored in conjunction with keypoint-based saliency maps for Focus-of-Attention. Recognition rates of up to 96% were achieved by combining frontal and 3/4 views, and recognition was quite robust against partial occlusions.

Blavigator: a navigation aid for blind persons

Blavigator (blind navigator) is a vision aid for blind and visuaIIy impaired persons. It supports local navigation by detecting waIkable paths in the immediate vicinity of the user. It guides the user for centering on the path.

Fast and accurate multi-scale keypoints based on end-stopped cells

Increasingly more applications in computer vision employ interest points. Algorithms like SIFT and SURF are all based on partial derivatives of images smoothed with Gaussian filter kemels. These algorithrns are fast and therefore very popular.

A disparity energy model improved by line, edge and keypoint correspondences

Disparity energy models (DEMs) estimate local depth information on the basis ofVl complex cells. Our recent DEM (Martins et al, 2011 ISSPlT261-266) employs a population code. Once the population's cells have been trained with randorn-dot stereograms, it is applied at all retinotopic positions in the visual field. Despite producing good results in textured regions, the model needs to be made more precise, especially at depth transitions.

Modelfest and contrast-interrelation-function data predicted by a retinal model

The automatic implementation of decoders for a visual perception is achieved as follows. The action described by a production rule is realized by means of the decoder in which a pattern of connections coreesponds to that of stimuli. According to "S.Karasawa,(Proc. of CCCT, Vol.5, pp.194-1999, Austin, Texas, August, 2004)", each program mable controllable connection among inputs is realized by a floating gate avalanche injection MOS FET, where inverted signals are used at writing, and the detection of matching between inputs and connections is carried out by using the signal source in which low level signal is provided via comparatively smaller resistance than high level.