A semantic-based approach to attain reproducibility of computational environments in scientific workflows: a case study

Reproducible research in scientific workflows is often addressed by tracking the provenance of the produced results. While this approach allows inspecting intermediate and final results, improves understanding, and permits replaying a workflow execution, it does not ensure that the computational environment is available for subsequent executions to reproduce the experiment. In this work, we propose describing the resources involved in the execution of an experiment using a set of semantic vocabularies, so as to conserve the computational environment. We define a process for documenting the workflow application, management system, and their dependencies based on 4 domain ontologies. We then conduct an experimental evaluation using a real workflow application on an academic and a public Cloud platform. Results show that our approach can reproduce an equivalent execution environment of a predefined virtual machine image on both computing platforms.

Standing together for reproducibility in large-scale computing: report on reproducibility@XSEDE

This is the final report on reproducibility@xsede, a one-day workshop held in conjunction with XSEDE14, the annual conference of the Extreme Science and Engineering Discovery Environment (XSEDE). The workshop's discussion-oriented agenda focused on reproducibility in large-scale computational research. Two important themes capture the spirit of the workshop submissions and discussions: (1) organizational stakeholders, especially supercomputer centers, are in a unique position to promote, enable, and support reproducible research; and (2) individual researchers should conduct each experiment as though someone will replicate that experiment. Participants documented numerous issues, questions, technologies, practices, and potentially promising initiatives emerging from the discussion, but also highlighted four areas of particular interest to XSEDE: (1) documentation and training that promotes reproducible research; (2) system-level tools that provide build- and run-time information at the level of the individual job; (3) the need to model best practices in research collaborations involving XSEDE staff; and (4) continued work on gateways and related technologies. In addition, an intriguing question emerged from the day's interactions: would there be value in establishing an annual award for excellence in reproducible research? Overview

Enhanced reproducibility of SADI web service workflows with Galaxy and Docke

Background: Semantic Web technologies have been widely applied in the life sciences, for example by data providers such as OpenLifeData and through web services frameworks such as SADI. The recently reported OpenLifeData2SADI project offers access to the vast OpenLifeData data store through SADI services. Findings: This article describes how to merge data retrieved from OpenLifeData2SADI with other SADI services using the Galaxy bioinformatics analysis platform, thus making this semantic data more amenable to complex analyses. This is demonstrated using a working example, which is made distributable and reproducible through a Docker image that includes SADI tools, along with the data and workflows that constitute the demonstration. Conclusions: The combination of Galaxy and Docker offers a solution for faithfully reproducing and sharing complex data retrieval and analysis workflows based on the SADI Semantic web service design patterns.

Repeatability, reproducibility, and accuracy of a novel pushbroom hyperspectral system

This study analyzes the repeatability, reproducibility and accuracy of a new hyperspectral system based on a pushbroom sensor as a means of measuring spectral features and color of materials and objects. The hyperspectral system consisted of a CCD camera, a spectrograph and an objective lens. An additional linear moving system allowed the mechanical scanning of the complete scene. A uniform overhead luminaire with daylight configuration was used to irradiate the scene using d:45 geometry. We followed the guidelines of the ASTM E2214-08 Standard Practice for Specifying and Verifying the Performance of Color-Measuring Instruments that define the standards and latest multidimensional procedures. The results obtained are analyzed in-depth and compared to those recently reported by other authors for spectrophotometers and multispectral systems. It can be concluded that hyperspectral systems are reliable and can be used in the industry to perform spectral and color readings with a high spatial resolution.

Reproducibility comparison among multiangle spectrophotometers

New color-measuring instruments known as multiangle spectrophotometers have been recently created to measure and characterize the goniochromism of special-effect pigments in many materials with a particular visual appearance (metallic, interference, pearlescent, sparkle, or glitter). These devices measure the gonioapparent color from the spectral relative reflectance factor and the L*a*b* values of the sample with different illumination and observation angles. These angles usually coincide with requirements marked in American Society for Testing and Materials (ASTM) and Deutsches Institut Für Normung standards relating to the gonioapparent color, but the results of comparisons between these instruments are still inconclusive. Therefore, the main purpose of this study is to compare several multiangle spectrophotometers at a reproducibility level according to ASTM E2214-08 guidelines. In particular, we compared two X-Rite multi-gonio spectrophotometers (MA98 and MA68II), a Datacolor multi-gonio spectrophotometer (FX10), and a BYK multi-gonio spectrophotometer (BYK-mac). These instruments share only five common measurement geometries: 45° × −30° (as 15°), 45° × −20° (as 25°), 45° × 0° (as 45°), 45° × 30° (as 75°), 45° × 65° (as 110°). Specific statistical studies were used for the reproducibility comparison, including a Hotelling test and a statistical intercomparison test to determine the confidence interval of the partial color differences ΔL*, Δa*, Δb*, and the total color difference ΔE*ab. This was conducted using a database collection of 88 metallic and pearlescent samples that were measured 20 times without the replacement of all the instruments. The final findings show that in most measurement geometries, the reproducibility differences between pairs of instruments are statistically significant, although in general, there is a better reproducibility level at certain common geometries for newer instruments (MA98 and BYK-mac). This means that these differences are due to systematic or bias errors (angle tolerances for each geometry, photometric scales, white standards, etc.), but not exclusively to random errors. However, neither of the statistical tests used is valid to discriminate and quantify the detected bias errors in this comparison between instruments.

Repeatability and reproducibility of corneal thickness using SOCT Copernicus HR

Purpose: The aim of this study is to determine the reliability of corneal thickness measurements derived from SOCT Copernicus HR (Fourier domain OCT). Methods: Thirty healthy eyes of 30 subjects were evaluated. One eye of each patient was chosen randomly. Images were obtained of the central (up to 2.0 mm from the corneal apex) and paracentral (2.0 to 4.0 mm) cornea. We assessed corneal thickness (central and paracentral) and epithelium thickness. The intra-observer repeatability data were analysed using the intra-class correlation coefficient (ICC) for a range of 95 per cent within-subject standard deviation (SW) and the within-subject coefficient of variation (CW). The level of agreement by Bland–Altman analysis was also represented for the study of the reproducibility between observers and agreement between methods of measurement (automatic versus manual). Results: The mean value of the central corneal thickness (CCT) was 542.4 ± 30.1 μm (SD). There was a high intra-observer agreement, finding the best result in the central sector with an intra-class correlation coefficient of 0.99, 95 per cent CI (0.989 to 0.997) and the worst, in the minimum corneal thickness, with an intra-class correlation coefficient of 0.672, 95 per cent CI (0.417 to 0.829). Reproducibility between observers was very high. The best result was found in the central sector thickness obtained both manually and automatically with an intra-class correlation coefficient of 0.990 in both cases and the worst result in the maximum corneal thickness with an intra-class correlation coefficient of 0.827. The agreement between measurement methods was also very high with intra-class correlation coefficient greater than 0.91. On the other hand the repeatability and reproducibility for epithelial measurements was poor. Conclusion: Pachymetric mapping with SOCT Copernicus HR was found to be highly repeatable and reproducible. We found that the device lacks an appropriate ergonomic design as proper focusing of the laser beam onto the cornea for anterior segment scanning required that patients were positioned slightly farther away from the machine head-rest than in the setup for retinal imaging.

Validation of standard surface reproducibility. Final report.

National Highway Traffic Safety Administration, Washington, D.C.

Reproducibility and accuracy of echocardiographic measurements of left ventricular parameters using real-time three-dimensional echocardiography

OBJECTIVES We sought to determine whether assessment of left ventricular (LV) function with real-time (RT) three-dimensional echocardiography (3DE) could reduce the variation of sequential LV measurements and provide greater accuracy than two-dimensional echocardiography (2DE). BACKGROUND Real-time 3DE has become feasible as a standard clinical tool, but its accuracy for LV assessment has not been validated. METHODS Unselected patients (n = 50; 41 men; age, 64 +/- 8 years) presenting for evaluation of LV function were studied with 2DE and RT-3DE. Test-retest variation was performed by a complete restudy by a separate sonographer within 1 h without alteration of hemodynamics or therapy. Magnetic resonance imaging (MRI) images were obtained during a breath-hold, and measurements were made off-line. RESULTS The test-retest variation showed similar measurements for volumes but wider scatter of LV mass measurements with M-mode and 2DE than 3DE. The average MRI end-diastolic volume was 172 +/- 53 ml; LV volumes were underestimated by 2DE (mean difference, -54 +/- 33; p < 0.01) but only slightly by RT-3DE (-4 +/- 29; p = 0.31). Similarly, end-systolic volume by MRI (91 +/- 53 ml) was underestimated by 2DE (mean difference, -28 +/- 28; p < 0.01) and by RT-3DE (mean difference, -3 +/- 18; p = 0.23). Ejection fraction by MRI was similar by 2DE (p = 0.76) and RT-3DE (p = 0.74). Left ventricular mass (183 +/- 50 g) was overestimated by M-mode (mean difference, 68 +/- 86 g; p < 0.01) and 2DE (16 +/- 57; p = 0.04) but not RT-3DE (0 +/- 38 g; p = 0.94). There was good inter- and intra-observer correlation between RT-3DE by two sonographers for volumes, ejection fraction, and mass. CONCLUSIONS Real-time 3DE is a feasible approach to reduce test-retest variation of LV volume, ejection fraction, and mass measurements in follow-up LV assessment in daily practice. (C) 2004 by the American College of Cardiology Foundation.

Reproducibility-repeatability of choroidal thickness calculation using optical coherence tomography

Purpose. To evaluate the repeatability and reproducibility of subfoveal choroidal thickness (CT) calculations performed manually using optical coherence tomography (OCT). Methods. The CT was imaged in vivo at each of two visits on 11 healthy volunteers (mean age, 35.72 ± 13.19 years) using the spectral domain OCT. CT was manually measured after applying ImageJ processing filters on 15 radial subfoveal scans. Each radial scan was spaced 12° from each other and contained 2500 A-scans. The coefficient of variability, coefficient of repeatability (CoR), coefficient of reproducibility, and intraclass correlation coefficient determined the reproducibility and repeatability of the calculation. Axial length (AL) and mean spherical equivalent refractive error were measured with the IOLMaster and an open view autorefractor to study their potential relationship with CT. Results. The within-visit and between-visit coefficient of variability, CoR, coefficient of reproducibility, and intraclass correlation coefficient were 0.80, 2.97% 2.44%, and 99%, respectively. The subfoveal CT correlated significantly with AL (R = -0.60, p = 0.05). Conclusions. The subfoveal CT could be measured manually in vivo using OCT and the readings obtained from the healthy subjects evaluated were repeatable and reproducible. It is proposed that OCT could be a useful instrument to perform in vivo assessment and monitoring of CT changes in retinal disease. The preliminary results suggest a negative correlation between subfoveal CT and AL in such a way that it decreases with increasing AL but not with refractive error.