Standardized evaluation of algorithms for computer-aided diagnosis of dementia based on structural MRI: The CADDementia challenge

Algorithms for computer-aided diagnosis of dementia based on structural MRI have demonstrated high performance in the literature, but are difficult to compare as different data sets and methodology were used for evaluation. In addition, it is unclear how the algorithms would perform on previously unseen data, and thus, how they would perform in clinical practice when there is no real opportunity to adapt the algorithm to the data at hand. To address these comparability, generalizability and clinical applicability issues, we organized a grand challenge that aimed to objectively compare algorithms based on a clinically representative multi-center data set. Using clinical practice as the starting point, the goal was to reproduce the clinical diagnosis. Therefore, we evaluated algorithms for multi-class classification of three diagnostic groups: patients with probable Alzheimer's disease, patients with mild cognitive impairment and healthy controls. The diagnosis based on clinical criteria was used as reference standard, as it was the best available reference despite its known limitations. For evaluation, a previously unseen test set was used consisting of 354 T1-weighted MRI scans with the diagnoses blinded. Fifteen research teams participated with a total of 29 algorithms. The algorithms were trained on a small training set (n = 30) and optionally on data from other sources (e.g., the Alzheimer's Disease Neuroimaging Initiative, the Australian Imaging Biomarkers and Lifestyle flagship study of aging). The best performing algorithm yielded an accuracy of 63.0% and an area under the receiver-operating-characteristic curve (AUC) of 78.8%. In general, the best performances were achieved using feature extraction based on voxel-based morphometry or a combination of features that included volume, cortical thickness, shape and intensity. The challenge is open for new submissions via the web-based framework: http://caddementia.grand-challenge.org.

Evaluation of two cloud parametrization schemes using ARM and Cloudnet observations

Ground-based remote-sensing observations from Atmospheric Radiation Measurement (ARM) and Cloud-Net sites are used to evaluate the clouds predicted by a weather forecasting and climate model. By evaluating the cloud predictions using separate measures for the errors in frequency of occurrence, amount when present, and timing, we provide a detailed assessment of the model performance, which is relevant to weather and climate time-scales. Importantly, this methodology will be of great use when attempting to develop a cloud parametrization scheme, as it provides a clearer picture of the current deficiencies in the predicted clouds. Using the Met Office Unified Model, it is shown that when cloud fractions produced by a diagnostic and a prognostic cloud scheme are compared, the prognostic cloud scheme shows improvements to the biases in frequency of occurrence of low, medium and high cloud and to the frequency distributions of cloud amount when cloud is present. The mean cloud profiles are generally improved, although it is shown that in some cases the diagnostic scheme produced misleadingly good mean profiles as a result of compensating errors in frequency of occurrence and amount when present. Some biases remain when using the prognostic scheme, notably the underprediction of mean ice cloud fraction due to the amount when present being too low, and the overprediction of mean liquid cloud fraction due to the frequency of occurrence being too high.

Fabrication and evaluation of a skutterudite-based thermoelectric module for high-temperature applications

We report a straightforward methodology for the fabrication of high-temperature thermoelectric (TE) modules using commercially available solder alloys and metal barriers. This methodology employs standard and accessible facilities that are simple to implement in any laboratory. A TE module formed by nine n-type Yb x Co4Sb12 and p-type Ce x Fe3CoSb12 state-of-the-art skutterudite material couples was fabricated. The physical properties of the synthesized skutterudites were determined, and the module power output, internal resistance, and thermocycling stability were evaluated in air. At a temperature difference of 365 K, the module provides more than 1.5 W cm−3 volume power density. However, thermocycling showed an increase of the internal module resistance and degradation in performance with the number of cycles when the device is operated at a hot-side temperature higher than 573 K. This may be attributed to oxidation of the skutterudite thermoelements.

An evaluation of the pedestrian classification in a multi-domain multi-modality setup

The objective of this article is to study the problem of pedestrian classification across different light spectrum domains (visible and far-infrared (FIR)) and modalities (intensity, depth and motion). In recent years, there has been a number of approaches for classifying and detecting pedestrians in both FIR and visible images, but the methods are difficult to compare, because either the datasets are not publicly available or they do not offer a comparison between the two domains. Our two primary contributions are the following: (1) we propose a public dataset, named RIFIR , containing both FIR and visible images collected in an urban environment from a moving vehicle during daytime; and (2) we compare the state-of-the-art features in a multi-modality setup: intensity, depth and flow, in far-infrared over visible domains. The experiments show that features families, intensity self-similarity (ISS), local binary patterns (LBP), local gradient patterns (LGP) and histogram of oriented gradients (HOG), computed from FIR and visible domains are highly complementary, but their relative performance varies across different modalities. In our experiments, the FIR domain has proven superior to the visible one for the task of pedestrian classification, but the overall best results are obtained by a multi-domain multi-modality multi-feature fusion.

Tracking performance evaluation on PETS 2015 Challenge datasets

This paper presents a quantitative evaluation of a tracking system on PETS 2015 Challenge datasets using well-established performance measures. Using the existing tools, the tracking system implements an end-to-end pipeline that include object detection, tracking and post- processing stages. The evaluation results are presented on the provided sequences of both ARENA and P5 datasets of PETS 2015 Challenge. The results show an encouraging performance of the tracker in terms of accuracy but a greater tendency of being prone to cardinality error and ID changes on both datasets. Moreover, the analysis show a better performance of the tracker on visible imagery than on thermal imagery.

Economic experiments used for the evaluation of building users’ energy-saving behavior

Different treatments that could be implemented in the home environ-ment are evaluated with the objective of reaching a more rational and efficient use of energy. We consider that a detailed knowledge of energy-consuming behaviour is paramount for the development and implementation of new technologies, services and even policies that could result in more rational energy use. The proposed evaluation methodology is based on the development of economic experiments implemented in an experimental economics laboratory, where the behaviour of individuals when making decisions related to energy use in the domestic environment can be tested.

Evaluation of CMIP5 palaeo-simulations to improve climate projections

Structural differences among models account for much of the uncertainty in projected climate changes, at least until the mid-twenty-first century. Recent observations encompass too limited a range of climate variability to provide a robust test of the ability to simulate climate changes. Past climate changes provide a unique opportunity for out-of-sample evaluation of model performance. Palaeo-evaluation has shown that the large-scale changes seen in twenty-first-century projections, including enhanced land–sea temperature contrast, latitudinal amplification, changes in temperature seasonality and scaling of precipitation with temperature, are likely to be realistic. Although models generally simulate changes in large-scale circulation sufficiently well to shift regional climates in the right direction, they often do not predict the correct magnitude of these changes. Differences in performance are only weakly related to modern-day biases or climate sensitivity, and more sophisticated models are not better at simulating climate changes. Although models correctly capture the broad patterns of climate change, improvements are required to produce reliable regional projections.

Evaluation of a modern-analogue methodology for reconstructing Australian palaeoclimate from pollen

Quantitative palaeoclimate reconstructions are widely used to evaluate climatemodel performance. Here, as part of an effort to provide such a data set for Australia, we examine the impact of analytical decisions and sampling assumptions on modern-analogue reconstructions using a continent-wide pollen data set. There is a high degree of correlation between temperature variables in the modern climate of Australia, but there is sufficient orthogonality in the variations of precipitation, summer and winter temperature and plant–available moisture to allow independent reconstructions of these four variables to be made. The method of analogue selection does not affect the reconstructions, although bootstrap resampling provides a more reliable technique for obtaining robust measures of uncertainty. The number of analogues used affects the quality of the reconstructions: the most robust reconstructions are obtained using 5 analogues. The quality of reconstructions based on post-1850 CE pollen samples differ little from those using samples from between 1450 and 1849 CE, showing that European post settlement modification of vegetation has no impact on the fidelity of the reconstructions although it substantially increases the availability of potential analogues. Reconstructions based on core top samples are more realistic than those using surface samples, but only using core top samples would substantially reduce the number of available analogues and therefore increases the uncertainty of the reconstructions. Spatial and/or temporal averaging of pollen assemblages prior to analysis negatively affects the subsequent reconstructions for some variables and increases the associated uncertainties. In addition, the quality of the reconstructions is affected by the degree of spatial smoothing of the original climate data, with the best reconstructions obtained using climate data froma 0.5° resolution grid, which corresponds to the typical size of the pollen catchment. This study provides a methodology that can be used to provide reliable palaeoclimate reconstructions for Australia, which will fill in a major gap in the data sets used to evaluate climate models.

Proper evaluation of the external convective heat transfer for the thermal analysis of cool roofs

Cool materials are characterized by high solar reflectance and high thermal emittance; when applied to the external surface of a roof, they make it possible to limit the amount of solar irradiance absorbed by the roof, and to increase the rate of heat flux emitted by irradiation to the environment, especially during nighttime. However, a roof also releases heat by convection on its external surface; this mechanism is not negligible, and an incorrect evaluation of its entity might introduce significant inaccuracy in the assessment of the thermal performance of a cool roof, in terms of surface temperature and rate of heat flux transferred to the indoors. This issue is particularly relevant in numerical simulations, which are essential in the design stage, therefore it deserves adequate attention. In the present paper, a review of the most common algorithms used for the calculation of the convective heat transfer coefficient due to wind on horizontal building surfaces is presented. Then, with reference to a case study in Italy, the simulated results are compared to the outcomes of a measurement campaign. Hence, the most appropriate algorithms for the convective coefficient are identified, and the errors deriving by an incorrect selection of this coefficient are discussed.

A process-based evaluation of dust-emitting winds in the CMIP5 simulation of HadGEM2-ES

Despite the importance of dust aerosol in the Earth system, state-of-the-art models show a large variety for North African dust emission. This study presents a systematic evaluation of dust emitting-winds in 30 years of the historical model simulation with the UK Met Office Earth-system model HadGEM2-ES for the Coupled Model Intercomparison Project Phase 5. Isolating the effect of winds on dust emission and using an automated detection for nocturnal low-level jets (NLLJs) allow an in-depth evaluation of the model performance for dust emission from a meteorological perspective. The findings highlight that NLLJs are a key driver for dust emission in HadGEM2-ES in terms of occurrence frequency and strength. The annually and spatially averaged occurrence frequency of NLLJs is similar in HadGEM2-ES and ERA-Interim from the European Centre for Medium-Range Weather Forecasts. Compared to ERA-Interim, a stronger pressure ridge over northern Africa in winter and the southward displaced heat low in summer result in differences in location and strength of NLLJs. Particularly the larger geostrophic winds associated with the stronger ridge have a strengthening effect on NLLJs over parts of West Africa in winter. Stronger NLLJs in summer may rather result from an artificially increased mixing coefficient under stable stratification that is weaker in HadGEM2-ES. NLLJs in the Bodélé Depression are affected by stronger synoptic-scale pressure gradients in HadGEM2-ES. Wintertime geostrophic winds can even be so strong that the associated vertical wind shear prevents the formation of NLLJs. These results call for further model improvements in the synoptic-scale dynamics and the physical parametrization of the nocturnal stable boundary layer to better represent dust-emitting processes in the atmospheric model. The new approach could be used for identifying systematic behavior in other models with respect to meteorological processes for dust emission. This would help to improve dust emission simulations and contribute to decreasing the currently large uncertainty in climate change projections with respect to dust aerosol.

Evaluation of thermal simulation of households in the metropolitan region of So Paulo, Brazil

This paper aims at assessing the performance of a program of thermal simulation (Arquitrop) in different households in the city of Sao Paulo, Brazil. The households were selected for the Wheezing Project which followed up children under 2 years old to monitor the occurrence of respiratory diseases. The results show that in all three study households there is a good approximation between the observed and the simulated indoor temperatures. It was also observed a fairly consistent and realistic behavior between the simulated indoor and the outdoor temperatures, describing the Arquitrop model as an efficient estimator and good representative of the thermal behavior of households in the city of Sao Paulo. The worst simulation is linked to the poorest type of construction. This may be explained by the bad quality of the construction, which the Architrop could not simulate adequately.

Evaluation of Recycled Tiles and OSB Ceiling Materials in Closed Broiler House Prototypes

This study aimed at evaluating the thermal performance of a modular ceiling system for poultry houses. The reduced- and distorted-scale prototypes used ceiling modules made of reforested wood and were covered with recycled long-life package tiles. The following parameters were measured for 21 days: the internal surface temperature (ST), globe temperature and humidity index (WBGT), and radiant heat load (RHL). Measurements were made at times of highest heat load (11:00 am, 13:00 pm, and 03:00 pm). Collected data were analyzed by ""R"" statistics software. Means were compared by multiple comparison test (Tukey) and linear regression was performed, both at 5% significance level. The results showed that the prototype with the ceiling was more efficient to reduce internal tile surface temperature; however, this was not sufficient to provide a comfortable environment for broilers during the growout. Therefore, other techniques to provide proper cooling are required in addition to the ceiling

Inconclusive results in conventional serological screening for Chagas` disease in blood banks: evaluation of cellular and humoral response

To find the most reliable screening method for Trypanosoma cruzi infection in blood banks. Epidemiological data, lymphoproliferation assay, parasitological, conventional serological tests: immunofluorescence, haemagglutination, ELISA with epimastigote and trypomastigote antigens and reference serological tests: trypomastigote excreted-secreted antigens (TESA) blot and chemiluminescent ELISA assay with mucine from trypomastigote forms were applied to individuals with inconclusive serology, non-chagasic individuals and chronic chagasic patients. TESA blot had the best performance when used as a single test in all the groups. In the inconclusive group 20.5% of individuals were positive for TESA blot, 23.3% for either lymphoproliferation or TESA blot, and 17.8% for lymphoproliferation only. Positive lymphoproliferation without detectable antibodies was observed in 5.47% of all inconclusive serology cases. Analysis of six parameters (three serological assays, at least one parasitological test, one lymphoproliferation assay and epidemiological data) in the inconclusive group showed that diagnosis of Chagas` disease was probable in 15 patients who were positive by two or more serological tests or for whom three of those six parameters were positive. TESA blot is a good confirmatory test for Chagas` disease in the inconclusive group. Although lymphoproliferation suggests the diagnosis of Chagas` disease in the absence of antibodies when associated with a high epidemiological risk of acquiring Chagas` disease, the data from this study and the characteristics of the lymphoproliferation assay (which is both laborious and time-consuming) do not support its use as a confirmatory test in blood-bank screening. However, our findings underscore the need to develop alternative methods that are not based on antibody detection to improve the diagnosis when serological tests are inconclusive.

A Survey on Graphical Methods for Classification Predictive Performance Evaluation

Predictive performance evaluation is a fundamental issue in design, development, and deployment of classification systems. As predictive performance evaluation is a multidimensional problem, single scalar summaries such as error rate, although quite convenient due to its simplicity, can seldom evaluate all the aspects that a complete and reliable evaluation must consider. Due to this, various graphical performance evaluation methods are increasingly drawing the attention of machine learning, data mining, and pattern recognition communities. The main advantage of these types of methods resides in their ability to depict the trade-offs between evaluation aspects in a multidimensional space rather than reducing these aspects to an arbitrarily chosen (and often biased) single scalar measure. Furthermore, to appropriately select a suitable graphical method for a given task, it is crucial to identify its strengths and weaknesses. This paper surveys various graphical methods often used for predictive performance evaluation. By presenting these methods in the same framework, we hope this paper may shed some light on deciding which methods are more suitable to use in different situations.

Synthesis, characterization and catalytic evaluation of cubic ordered mesoporous iron-silicon oxides

Iron was successfully incorporated in FDU-1 type cubic ordered mesoporous silica by a simple direct synthesis route. The (Fe/FDU-1) samples were characterized by Rutherford back-scattering spectrometry (RBS), small angle X-ray scattering (SAXS). N(2) sorption isotherm, X-ray diffraction (XRD) and X-ray absorption spectroscopy (XAS). The resulting material presented an iron content of about 5%. Prepared at the usual acid pH of -0.3, the composite was mostly formed by amorphous silica and hematite with a quantity of Fe(2+) present in the structure. The samples prepared with adjusted pH values (2 and 3.5) were amorphous. The samples` average pore diameter was around 12.0 nm and BET specific surface area was of 680 m(2) g(-1). Although the iron-incorporated material presented larger lattice parameter, about 25 nm compared to pure FDU-1, the Fe/FDU-1 composite still maintained its cubic ordered fcc mesoporous structure before and after the template removal at 540 degrees C. The catalytic performance of Fe/FDU-1 was investigated in the catalytic oxidation of Black Remazol B dye using a catalytic ozonation process. The results indicated that Fe/FDU-1 prepared at the usual acid pH exhibited high catalytic activity in the mineralization of this pollutant when compared to the pure FDU-1. Fe(2)O(3) and Fe/FDU-1 prepared with higher pH of 2 and 3.5. (C) 2010 Elsevier B.V. All rights reserved.