Reproducibility and accuracy of linear measurements on dental models derived from cone-beam computed tomography compared with digital dental casts.

INTRODUCTION The aim of this study was to determine the reproducibility and accuracy of linear measurements on 2 types of dental models derived from cone-beam computed tomography (CBCT) scans: CBCT images, and Anatomodels (InVivoDental, San Jose, Calif); these were compared with digital models generated from dental impressions (Digimodels; Orthoproof, Nieuwegein, The Netherlands). The Digimodels were used as the reference standard. METHODS The 3 types of digital models were made from 10 subjects. Four examiners repeated 37 linear tooth and arch measurements 10 times. Paired t tests and the intraclass correlation coefficient were performed to determine the reproducibility and accuracy of the measurements. RESULTS The CBCT images showed significantly smaller intraclass correlation coefficient values and larger duplicate measurement errors compared with the corresponding values for Digimodels and Anatomodels. The average difference between measurements on CBCT images and Digimodels ranged from -0.4 to 1.65 mm, with limits of agreement values up to 1.3 mm for crown-width measurements. The average difference between Anatomodels and Digimodels ranged from -0.42 to 0.84 mm with limits of agreement values up to 1.65 mm. CONCLUSIONS Statistically significant differences between measurements on Digimodels and Anatomodels, and between Digimodels and CBCT images, were found. Although the mean differences might be clinically acceptable, the random errors were relatively large compared with corresponding measurements reported in the literature for both Anatomodels and CBCT images, and might be clinically important. Therefore, with the CBCT settings used in this study, measurements made directly on CBCT images and Anatomodels are not as accurate as measurements on Digimodels.

Reproducibility of peripheral arterial tonometry measurements in male cardiovascular patients

BACKGROUND Assessment of endothelial function of the microvasculature by peripheral arterial tonometry (EndoPAT(®)) has gained increasing popularity in patients with cardiovascular risk factors. Only limited knowledge about its reproducibility in patients with coronary artery disease (CAD) is available. We therefore aimed to quantify reproducibility of EndoPAT(®) parameters in patients with stable CAD. DESIGN EndoPAT(®) measurements were performed repeatedly in 78 male patients (age 66 ± 8 years) with CAD on stable medication. We calculated overall mean, standard deviation (SD), coefficient of variation (CV) and intraclass correlation coefficient (ICC) of the following parameters: reactive hyperemic index (RHI), PAT ratio of the postocclusion period 90-150 s as used for calculation of the RHI (PAT ratio90-150 s) and 90-120 s (PAT ratio90-120 s) as used for the often employed Framingham RHI (F-RHI), as well as PAT ratio of the peak hyperemic response (PAT ratiopeak response). Additionally, least significant changes (LSC) for individual subjects and minimum sample sizes for parallel and cross-over design studies were calculated. RESULTS Mean RHI was 1·84 (SD 0·36). For RHI, PAT ratio90-150 s , PAT ratio90-120 s , and PAT ratiopeak response the CVs were 17·0%, 25·4%, 26·1%, and 25·0%, respectively. The ICCs were 0·45, 0·49, 0·48 and 0·51, respectively, and LSC for RHI was 47·2%. CONCLUSIONS CV of RHI in our population was moderate; however, we consider this precision insufficient to monitor changes in individual patients, as they would need to exceed 47% to show a significant change. Further, the poor ICCs reflect the difficulty of detecting treatment effects in homogenous populations, such as patients with stable CAD.

Loss on ignition as a method for estimating organic and carbonate content in sediments: reproducibility and comparability of results

Five test runs were performed to assess possible bias when performing the loss on ignition (LOI) method to estimate organic matter and carbonate content of lake sediments. An accurate and stable weight loss was achieved after 2 h of burning pure CaCO3 at 950 °C, whereas LOI of pure graphite at 530 °C showed a direct relation to sample size and exposure time, with only 40-70% of the possible weight loss reached after 2 h of exposure and smaller samples losing weight faster than larger ones. Experiments with a standardised lake sediment revealed a strong initial weight loss at 550 °C, but samples continued to lose weight at a slow rate at exposure of up to 64 h, which was likely the effect of loss of volatile salts, structural water of clay minerals or metal oxides, or of inorganic carbon after the initial burning of organic matter. A further test-run revealed that at 550 °C samples in the centre of the furnace lost more weight than marginal samples. At 950 °C this pattern was still apparent but the differences became negligible. Again, LOI was dependent on sample size. An analytical LOI quality control experiment including ten different laboratories was carried out using each laboratory's own LOI procedure as well as a standardised LOI procedure to analyse three different sediments. The range of LOI values between laboratories measured at 550 °C was generally larger when each laboratory used its own method than when using the standard method. This was similar for 950 °C, although the range of values tended to be smaller. The within-laboratory range of LOI measurements for a given sediment was generally small. Comparisons of the results of the individual and the standardised method suggest that there is a laboratory-specific pattern in the results, probably due to differences in laboratory equipment and/or handling that could not be eliminated by standardising the LOI procedure. Factors such as sample size, exposure time, position of samples in the furnace and the laboratory measuring affected LOI results, with LOI at 550 °C being more susceptible to these factors than LOI at 950 °C. We, therefore, recommend analysts to be consistent in the LOI method used in relation to the ignition temperatures, exposure times, and the sample size and to include information on these three parameters when referring to the method.

Good reproducibility of heart rate variability after orthostatic challenge in patients with a history of acute coronary syndrome

BACKGROUND: Several parameters of heart rate variability (HRV) have been shown to predict the risk of sudden cardiac death (SCD) in cardiac patients. There is consensus that risk prediction is increased when measuring HRV during specific provocations such as orthostatic challenge. For the first time, we provide data on reproducibility of such a test in patients with a history of acute coronary syndrome. METHODS: Sixty male patients (65+/-8years) with a history of acute coronary syndrome on stable medication were included. HRV was measured in supine (5min) and standing (5min) position on 2 occasions separated by two weeks. For risk assessment relevant time-domain [standard deviation of all R-R intervals (SDNN) and root mean squared standard differences between adjacent R-R intervals (RMSSD)], frequency domain [low-frequency power (LF), high-frequency power (HF) and LF/HF power ratio] and short-term fractal scaling component (DF1) were computed. Absolute reproducibility was assessed with the standard errors of the mean (SEM) and 95% limits of random variation, and relative reproducibility by the intraclass correlation coefficient (ICC). RESULTS: We found comparable SEMs and ICCs in supine position and after an orthostatic challenge test. All ICCs were good to excellent (ICCs between 0.636 and 0.869). CONCLUSIONS: Reproducibility of HRV parameters during orthostatic challenge is good and comparable with supine position.

Evaluation of New Cone-beam Computed Tomographic Criteria for Radiographic Healing Evaluation after Apical Surgery: Assessment of Repeatability and Reproducibility.

INTRODUCTION Conventional 2-dimensional radiography uses defined criteria for outcome assessment of apical surgery. However, these radiographic healing criteria are not applicable for 3-dimensional radiography. The present study evaluated the repeatability and reproducibility of new cone-beam computed tomographic (CBCT)-based healing criteria for the judgment of periapical healing 1 year after apical surgery. METHODS CBCT scans taken 1 year after apical surgery (61 roots of 54 teeth in 54 patients, mean age = 54.4 years) were evaluated by 3 blinded and calibrated observers using 4 different indices. Reformatted buccolingual CBCT sections through the longitudinal axis of the treated roots were analyzed. Radiographic healing was assessed at the resection plane (R index), within the apical area (A index), of the cortical plate (C index), and regarding a combined apical-cortical area (B index). All readings were performed twice to calculate the intraobserver agreement (repeatability). Second-time readings were used for analyzing the interobserver agreement (reproducibility). Various statistical tests (Cohen, kappa, Fisher, and Spearman) were performed to measure the intra- and interobserver concurrence, the variability of score ratios, and the correlation of indices. RESULTS For all indices, the rates of identical first- and second-time scores were always higher than 80% (intraobserver Cohen κ values ranging from 0.793 to 0.963). The B index (94.0%) showed the highest intraobserver agreement. Regarding interobserver agreement, the highest rate was found for the B index (72.1%). The Fleiss' κ values for R and B indices exhibited substantial agreement (0.626 and 0.717, respectively), whereas the values for A and C indices showed moderate agreement (0.561 and 0.573, respectively). The Spearman correlation coefficients for R, A, C, and B indices all exhibited a moderate to very strong correlation with the highest correlation found between C and B indices (rs = 0.8069). CONCLUSIONS All indices showed an excellent intraobserver agreement (repeatability). With regard to interobserver agreement (reproducibility), the B index (healing of apical and cortical defects combined) and the R index (healing on the resection plane) showed substantial congruence and thus are to be recommended in future studies when using buccolingual CBCT sections for radiographic outcome assessment of apical surgery.

Quantifying Reproducibility in Computational Biology: The Case of the Tuberculosis Drugome

How easy is it to reproduce the results found in a typical computational biology paper? Either through experience or intuition the reader will already know that the answer is with difficulty or not at all. In this paper we attempt to quantify this difficulty by reproducing a previously published paper for different classes of users (ranging from users with little expertise to domain experts) and suggest ways in which the situation might be improved. Quantification is achieved by estimating the time required to reproduce each of the steps in the method described in the original paper and make them part of an explicit workflow that reproduces the original results. Reproducing the method took several months of effort, and required using new versions and new software that posed challenges to reconstructing and validating the results. The quantification leads to “reproducibility maps” that reveal that novice researchers would only be able to reproduce a few of the steps in the method, and that only expert researchers with advance knowledge of the domain would be able to reproduce the method in its entirety. The workflow itself is published as an online resource together with supporting software and data. The paper concludes with a brief discussion of the complexities of requiring reproducibility in terms of cost versus benefit, and a desiderata with our observations and guidelines for improving reproducibility. This has implications not only in reproducing the work of others from published papers, but reproducing work from one’s own laboratory.

Leveraging Semantics to Improve Reproducibility in Scientific Workflows

Reproducibility of published results is a cornerstone in scientific publishing and progress. Therefore, the scientific community has been encouraging authors and editors to publish their contributions in a verifiable and understandable way. Efforts such as the Reproducibility Initiative [1], or the Reproducibility Projects on Biology [2] and Psychology [3] domains, have been defining standards and patterns to assess whether an experimental result is reproducible.

A Semantic-Based Approach to Attain Reproducibility of Computational Environments in Scientic Work ows: A Case Study

Reproducible research in scientic work ows is often addressed by tracking the provenance of the produced results. While this approach allows inspecting intermediate and nal results, improves understanding, and permits replaying a work ow execution, it does not ensure that the computational environment is available for subsequent executions to reproduce the experiment. In this work, we propose describing the resources involved in the execution of an experiment using a set of semantic vocabularies, so as to conserve the computational environment. We dene a process for documenting the work ow application, management system, and their dependencies based on 4 domain ontologies. We then conduct an experimental evaluation sing a real work ow application on an academic and a public Cloud platform. Results show that our approach can reproduce an equivalent execution environment of a predened virtual machine image on both computing platforms.

A semantic-based approach to attain reproducibility of computational environments in scientific workflows: a case study

Reproducible research in scientific workflows is often addressed by tracking the provenance of the produced results. While this approach allows inspecting intermediate and final results, improves understanding, and permits replaying a workflow execution, it does not ensure that the computational environment is available for subsequent executions to reproduce the experiment. In this work, we propose describing the resources involved in the execution of an experiment using a set of semantic vocabularies, so as to conserve the computational environment. We define a process for documenting the workflow application, management system, and their dependencies based on 4 domain ontologies. We then conduct an experimental evaluation using a real workflow application on an academic and a public Cloud platform. Results show that our approach can reproduce an equivalent execution environment of a predefined virtual machine image on both computing platforms.

Standing together for reproducibility in large-scale computing: report on reproducibility@XSEDE

This is the final report on reproducibility@xsede, a one-day workshop held in conjunction with XSEDE14, the annual conference of the Extreme Science and Engineering Discovery Environment (XSEDE). The workshop's discussion-oriented agenda focused on reproducibility in large-scale computational research. Two important themes capture the spirit of the workshop submissions and discussions: (1) organizational stakeholders, especially supercomputer centers, are in a unique position to promote, enable, and support reproducible research; and (2) individual researchers should conduct each experiment as though someone will replicate that experiment. Participants documented numerous issues, questions, technologies, practices, and potentially promising initiatives emerging from the discussion, but also highlighted four areas of particular interest to XSEDE: (1) documentation and training that promotes reproducible research; (2) system-level tools that provide build- and run-time information at the level of the individual job; (3) the need to model best practices in research collaborations involving XSEDE staff; and (4) continued work on gateways and related technologies. In addition, an intriguing question emerged from the day's interactions: would there be value in establishing an annual award for excellence in reproducible research? Overview

Enhanced reproducibility of SADI web service workflows with Galaxy and Docke

Background: Semantic Web technologies have been widely applied in the life sciences, for example by data providers such as OpenLifeData and through web services frameworks such as SADI. The recently reported OpenLifeData2SADI project offers access to the vast OpenLifeData data store through SADI services. Findings: This article describes how to merge data retrieved from OpenLifeData2SADI with other SADI services using the Galaxy bioinformatics analysis platform, thus making this semantic data more amenable to complex analyses. This is demonstrated using a working example, which is made distributable and reproducible through a Docker image that includes SADI tools, along with the data and workflows that constitute the demonstration. Conclusions: The combination of Galaxy and Docker offers a solution for faithfully reproducing and sharing complex data retrieval and analysis workflows based on the SADI Semantic web service design patterns.

Repeatability, reproducibility, and accuracy of a novel pushbroom hyperspectral system

This study analyzes the repeatability, reproducibility and accuracy of a new hyperspectral system based on a pushbroom sensor as a means of measuring spectral features and color of materials and objects. The hyperspectral system consisted of a CCD camera, a spectrograph and an objective lens. An additional linear moving system allowed the mechanical scanning of the complete scene. A uniform overhead luminaire with daylight configuration was used to irradiate the scene using d:45 geometry. We followed the guidelines of the ASTM E2214-08 Standard Practice for Specifying and Verifying the Performance of Color-Measuring Instruments that define the standards and latest multidimensional procedures. The results obtained are analyzed in-depth and compared to those recently reported by other authors for spectrophotometers and multispectral systems. It can be concluded that hyperspectral systems are reliable and can be used in the industry to perform spectral and color readings with a high spatial resolution.

Reproducibility comparison among multiangle spectrophotometers

New color-measuring instruments known as multiangle spectrophotometers have been recently created to measure and characterize the goniochromism of special-effect pigments in many materials with a particular visual appearance (metallic, interference, pearlescent, sparkle, or glitter). These devices measure the gonioapparent color from the spectral relative reflectance factor and the L*a*b* values of the sample with different illumination and observation angles. These angles usually coincide with requirements marked in American Society for Testing and Materials (ASTM) and Deutsches Institut Für Normung standards relating to the gonioapparent color, but the results of comparisons between these instruments are still inconclusive. Therefore, the main purpose of this study is to compare several multiangle spectrophotometers at a reproducibility level according to ASTM E2214-08 guidelines. In particular, we compared two X-Rite multi-gonio spectrophotometers (MA98 and MA68II), a Datacolor multi-gonio spectrophotometer (FX10), and a BYK multi-gonio spectrophotometer (BYK-mac). These instruments share only five common measurement geometries: 45° × −30° (as 15°), 45° × −20° (as 25°), 45° × 0° (as 45°), 45° × 30° (as 75°), 45° × 65° (as 110°). Specific statistical studies were used for the reproducibility comparison, including a Hotelling test and a statistical intercomparison test to determine the confidence interval of the partial color differences ΔL*, Δa*, Δb*, and the total color difference ΔE*ab. This was conducted using a database collection of 88 metallic and pearlescent samples that were measured 20 times without the replacement of all the instruments. The final findings show that in most measurement geometries, the reproducibility differences between pairs of instruments are statistically significant, although in general, there is a better reproducibility level at certain common geometries for newer instruments (MA98 and BYK-mac). This means that these differences are due to systematic or bias errors (angle tolerances for each geometry, photometric scales, white standards, etc.), but not exclusively to random errors. However, neither of the statistical tests used is valid to discriminate and quantify the detected bias errors in this comparison between instruments.

Repeatability and reproducibility of corneal thickness using SOCT Copernicus HR

Purpose: The aim of this study is to determine the reliability of corneal thickness measurements derived from SOCT Copernicus HR (Fourier domain OCT). Methods: Thirty healthy eyes of 30 subjects were evaluated. One eye of each patient was chosen randomly. Images were obtained of the central (up to 2.0 mm from the corneal apex) and paracentral (2.0 to 4.0 mm) cornea. We assessed corneal thickness (central and paracentral) and epithelium thickness. The intra-observer repeatability data were analysed using the intra-class correlation coefficient (ICC) for a range of 95 per cent within-subject standard deviation (SW) and the within-subject coefficient of variation (CW). The level of agreement by Bland–Altman analysis was also represented for the study of the reproducibility between observers and agreement between methods of measurement (automatic versus manual). Results: The mean value of the central corneal thickness (CCT) was 542.4 ± 30.1 μm (SD). There was a high intra-observer agreement, finding the best result in the central sector with an intra-class correlation coefficient of 0.99, 95 per cent CI (0.989 to 0.997) and the worst, in the minimum corneal thickness, with an intra-class correlation coefficient of 0.672, 95 per cent CI (0.417 to 0.829). Reproducibility between observers was very high. The best result was found in the central sector thickness obtained both manually and automatically with an intra-class correlation coefficient of 0.990 in both cases and the worst result in the maximum corneal thickness with an intra-class correlation coefficient of 0.827. The agreement between measurement methods was also very high with intra-class correlation coefficient greater than 0.91. On the other hand the repeatability and reproducibility for epithelial measurements was poor. Conclusion: Pachymetric mapping with SOCT Copernicus HR was found to be highly repeatable and reproducible. We found that the device lacks an appropriate ergonomic design as proper focusing of the laser beam onto the cornea for anterior segment scanning required that patients were positioned slightly farther away from the machine head-rest than in the setup for retinal imaging.