Comparing predictions of extinction risk using models and subjective judgement

Models of population dynamics are commonly used to predict risks in ecology, particularly risks of population decline. There is often considerable uncertainty associated with these predictions. However, alternatives to predictions based on population models have not been assessed. We used simulation models of hypothetical species to generate the kinds of data that might typically be available to ecologists and then invited other researchers to predict risks of population declines using these data. The accuracy of the predictions was assessed by comparison with the forecasts of the original model. The researchers used either population models or subjective judgement to make their predictions. Predictions made using models were only slightly more accurate than subjective judgements of risk. However, predictions using models tended to be unbiased, while subjective judgements were biased towards over-estimation. Psychology literature suggests that the bias of subjective judgements is likely to vary somewhat unpredictably among people, depending on their stake in the outcome. This will make subjective predictions more uncertain and less transparent than those based on models. (C) 2004 Elsevier SAS. All rights reserved.

Initial placement methods for Haitian adult ESOL students : the NYS place test plus profile data compared to subjective assessment

Few valid and reliable placement procedures are available to assess the English language proficiency of adults who enroll in English for Speakers of Other Languages (ESOL) programs. Whereas placement material exists for children and university ESOL students, the needs of students in adult community education programs have not been adequately addressed. Furthermore, the research suggests that a number of variables, such as, native language, age, prior schooling, length of residence, and employment are related to second language acquisition. Numerous studies contribute to our understanding of the relationship of these factors to second language acquisition of Spanish-speaking students. Again, there is a void in the research investigating the factors affecting second language acquisition and consequently, appropriate placement of Haitian Creole-speaking students. This study compared a standardized instrument, the NYS Place Test, used alone and in combination with a writing sample in English, to subjective judgement of a department coordinator for initial placement of Haitian adult ESOL students in a community education program. The study also investigated whether or not consideration of student profile data improved the accuracy of the test. Finally, the study sought to determine if a relationship existed between student profile data and those who withdrew from the program or did not enter a class after registering. Analysis of the data by crosstabulation and chi-square revealed that the standardized NYS Place Test was at least as accurate as subjective department coordinator placement and that one procedure could be substituted for li other. Although the writing sample in English improved accuracy of placement by the NYS test, the results were not significant. Of the profile variables, only length of residence was found to be significantly related to accuracy of placement using the NYS Place Test. The number of incorrect placements was higher for those students who lived in the host country from twenty-five to one hundred ten months. A post hoc analysis of NYS test scores according to level showed that those learners who placed in level three also had a significantly higher incidence of incorrect placements. No significant relationship was observed between the profile variables and those who withdrew from the program or registered but did not enter a class.

Inter-rater agreement and reliability of the COSMIN (COnsensus-based Standards for the selection of health status Measurement Instruments) Checklist

Background: The COSMIN checklist is a tool for evaluating the methodological quality of studies on measurement properties of health-related patient-reported outcomes. The aim of this study is to determine the inter-rater agreement and reliability of each item score of the COSMIN checklist (n = 114). Methods: 75 articles evaluating measurement properties were randomly selected from the bibliographic database compiled by the Patient-Reported Outcome Measurement Group, Oxford, UK. Raters were asked to assess the methodological quality of three articles, using the COSMIN checklist. In a one-way design, percentage agreement and intraclass kappa coefficients or quadratic-weighted kappa coefficients were calculated for each item. Results: 88 raters participated. Of the 75 selected articles, 26 articles were rated by four to six participants, and 49 by two or three participants. Overall, percentage agreement was appropriate (68% was above 80% agreement), and the kappa coefficients for the COSMIN items were low (61% was below 0.40, 6% was above 0.75). Reasons for low inter-rater agreement were need for subjective judgement, and accustom to different standards, terminology and definitions.Conclusions: Results indicated that raters often choose the same response option, but that it is difficult on item level to distinguish between articles. When using the COSMIN checklist in a systematic review, we recommend getting some training and experience, completing it by two independent raters, and reaching consensus on one final rating. Instructions for using the checklist are improved.

Development of quantitative methods for sustainability assessment in process industry

The increasing aversion to technological risks of the society requires the development of inherently safer and environmentally friendlier processes, besides assuring the economic competitiveness of the industrial activities. The different forms of impact (e.g. environmental, economic and societal) are frequently characterized by conflicting reduction strategies and must be holistically taken into account in order to identify the optimal solutions in process design. Though the literature reports an extensive discussion of strategies and specific principles, quantitative assessment tools are required to identify the marginal improvements in alternative design options, to allow the trade-off among contradictory aspects and to prevent the “risk shift”. In the present work a set of integrated quantitative tools for design assessment (i.e. design support system) was developed. The tools were specifically dedicated to the implementation of sustainability and inherent safety in process and plant design activities, with respect to chemical and industrial processes in which substances dangerous for humans and environment are used or stored. The tools were mainly devoted to the application in the stages of “conceptual” and “basic design”, when the project is still open to changes (due to the large number of degrees of freedom) which may comprise of strategies to improve sustainability and inherent safety. The set of developed tools includes different phases of the design activities, all through the lifecycle of a project (inventories, process flow diagrams, preliminary plant lay-out plans). The development of such tools gives a substantial contribution to fill the present gap in the availability of sound supports for implementing safety and sustainability in early phases of process design. The proposed decision support system was based on the development of a set of leading key performance indicators (KPIs), which ensure the assessment of economic, societal and environmental impacts of a process (i.e. sustainability profile). The KPIs were based on impact models (also complex), but are easy and swift in the practical application. Their full evaluation is possible also starting from the limited data available during early process design. Innovative reference criteria were developed to compare and aggregate the KPIs on the basis of the actual sitespecific impact burden and the sustainability policy. Particular attention was devoted to the development of reliable criteria and tools for the assessment of inherent safety in different stages of the project lifecycle. The assessment follows an innovative approach in the analysis of inherent safety, based on both the calculation of the expected consequences of potential accidents and the evaluation of the hazards related to equipment. The methodology overrides several problems present in the previous methods proposed for quantitative inherent safety assessment (use of arbitrary indexes, subjective judgement, build-in assumptions, etc.). A specific procedure was defined for the assessment of the hazards related to the formations of undesired substances in chemical systems undergoing “out of control” conditions. In the assessment of layout plans, “ad hoc” tools were developed to account for the hazard of domino escalations and the safety economics. The effectiveness and value of the tools were demonstrated by the application to a large number of case studies concerning different kinds of design activities (choice of materials, design of the process, of the plant, of the layout) and different types of processes/plants (chemical industry, storage facilities, waste disposal). An experimental survey (analysis of the thermal stability of isomers of nitrobenzaldehyde) provided the input data necessary to demonstrate the method for inherent safety assessment of materials.

Capturing expert knowledge for threatened species assessments: a case study using NatureServe conservation status ranks

Assessments for assigning the conservation status of threatened species that are based purely on subjective judgements become problematic because assessments can be influenced by hidden assumptions, personal biases and perceptions of risks, making the assessment process difficult to repeat. This can result in inconsistent assessments and misclassifications, which can lead to a lack of confidence in species assessments. It is almost impossible to Understand an expert's logic or visualise the underlying reasoning behind the many hidden assumptions used throughout the assessment process. In this paper, we formalise the decision making process of experts, by capturing their logical ordering of information, their assumptions and reasoning, and transferring them into a set of decisions rules. We illustrate this through the process used to evaluate the conservation status of species under the NatureServe system (Master, 1991). NatureServe status assessments have been used for over two decades to set conservation priorities for threatened species throughout North America. We develop a conditional point-scoring method, to reflect the current subjective process. In two test comparisons, 77% of species' assessments using the explicit NatureServe method matched the qualitative assessments done subjectively by NatureServe staff. Of those that differed, no rank varied by more than one rank level under the two methods. In general, the explicit NatureServe method tended to be more precautionary than the subjective assessments. The rank differences that emerged from the comparisons may be due, at least in part, to the flexibility of the qualitative system, which allows different factors to be weighted on a species-by-species basis according to expert judgement. The method outlined in this study is the first documented attempt to explicitly define a transparent process for weighting and combining factors under the NatureServe system. The process of eliciting expert knowledge identifies how information is combined and highlights any inconsistent logic that may not be obvious in Subjective decisions. The method provides a repeatable, transparent, and explicit benchmark for feedback, further development, and improvement. (C) 2004 Elsevier SAS. All rights reserved.

Protocols for listing threatened species can forecast extinction

Risk-ranking protocols are used widely to classify the conservation status of the world's species. Here we report on the first empirical assessment of their reliability by using a retrospective study of 18 pairs of bird and mammal species (one species extinct and the other extant) with eight different assessors. The performance of individual assessors varied substantially, but performance was improved by incorporating uncertainty in parameter estimates and consensus among the assessors. When this was done, the ranks from the protocols were consistent with the extinction outcome in 70-80% of pairs and there were mismatches in only 10-20% of cases. This performance was similar to the subjective judgements of the assessors after they had estimated the range and population parameters required by the protocols, and better than any single parameter. When used to inform subjective judgement, the protocols therefore offer a means of reducing unpredictable biases that may be associated with expert input and have the advantage of making the logic behind assessments explicit. We conclude that the protocols are useful for forecasting extinctions, although they are prone to some errors that have implications for conservation. Some level of error is to be expected, however, given the influence of chance on extinction. The performance of risk assessment protocols may be improved by providing training in the application of the protocols, incorporating uncertainty in parameter estimates and using consensus among multiple assessors, including some who are experts in the application of the protocols. Continued testing and refinement of the protocols may help to provide better absolute estimates of risk, particularly by re-evaluating how the protocols accommodate missing data.

The problem of appraising qualitative research

Qualitative research can make a valuable contribution to the study of quality and safety in health care. Sound ways of appraising qualitative research are needed, but currently there are many different proposals with few signs of an emerging consensus. One problem has been the tendency to treat qualitative research as a unified field. We distinguish universal features of quality from those specific to methodology and offer a set of minimally prescriptive prompts to assist with the assessment of generic features of qualitative research. In using these, account will need to be taken of the particular method of data collection and methodological approach being used. There may be a need for appraisal criteria suited to the different methods of qualitative data collection and to different methodological approaches. These more specific criteria would help to distinguish fatal flaws from more minor errors in the design, conduct, and reporting of qualitative research. There will be difficulties in doing this because some aspects of qualitative research, particularly those relating to quality of insight and interpretation, will remain difficult to appraise and will rely largely on subjective judgement.

STATIC IMAGES WITH DIFFERENT INDUCED INTENSITIES OF HUMAN BODY MOVEMENTS AFFECT SUBJECTIVE TIME

Modulation of subjective time was examined using static images eliciting perceptions of different intensities of body movement. Undergraduate students were exposed to photographs of dancer sculptures in different dance positions for 36 sec. and asked to estimate the exposure duration. Lower movement intensities were related to shorter estimated durations. Mean durations for images of unmoving dancers were underestimated and for dancers taking a ballet step were overestimated. Temporal estimations were also related to the order of presentation of the stimuli, which suggested that subjective time estimations were influenced by the experimental context. Subjective time is related not only to the visual perception of moving images, but also of elicited perceptions of movement in static images, suggesting an embodiment effect on subjective time estimation.

Tonal Modulation and Subjective Time

This work investigated listeners` sense of the temporal expression of tonal modulation. One experiment described the effects on retrospective reproductions of sudden and gradual modulations to close and distant keys. The results showed that modulations elicit time underestimations as an inverse function of interkey distances, with a major impact for sudden modulations. A proposed vectorial model - ""Expected Development Fraction"" (EDF) - describes the development of expectations when an interkey distance is traversed during a certain time interval. This expected development is longer than the perceived duration, leading to underestimation of the time.

The tannin-degrading species Streptococcus gallolyticus and Streptococcus caprinus are subjective synonyms

The tannin-degrading species Streptococcus gallolyticus and Streptococcus caprinus have been shown to be subjective synonyms on the basis of their levels of 16S rRNA sequence similarity (98.3%) and DNA-DNA homology (>70%) and the phenotypes of their type strains. S. gallolyticus has nomenclatural priority according to Rule 24b(2) of the International Code of Nomenclature of Bacteria.

Subjective memory impairment in a rural population with low education in the Amazon rainforest: an exploratory study

Background: The high prevalence of subjective memory impairment (SMI) in the elderly living in developed countries may be partly dependent on greater demand placed on them by new technologies. As part of a comprehensive study on cognitive impairment in a population living in the Amazon rainforest, we evaluated the prevalence of SMI and investigated the features associated with it. Methods: We evaluated 163 subjects (82 females) with a mean age of 62.3 years (50-94 years), 110 of whom were illiterate, using the answer to a single question ""Do you have memory problems?"" to classify them into groups with or without SMI. The assessment involved application of the Mini-mental State Examination (MMSE), delayed recall from the Brief Cognitive Battery designed for the evaluation of low educated and illiterate individuals, the Patient Questionnaire (PQ) of the Primary Care Evaluation of Mental Disorders (PRIME-MD), and the Happiness Analogical Scale. Results: A very high prevalence of SMI (70%) was observed, exceeding rates reported by similar studies conducted in developed countries. SMI was more frequent in women, whereas age and education did not impact on prevalence. Subjects with SMI had significantly more somatic and psychiatric symptoms on the PQ, as well as lower means on the MMSE, but not on the delayed recall test. Multiple logistic regressions showed that the most important factor associated with the presence of SMI was a high score on the PQ (OR: 3.84, p = 0.011). Conclusion: Psychological and somatic symptoms may be the principal cause of SMI in this population.

Comparison of Quantitative Imaging Devices and Subjective Optic Nerve Head Assessment by General Ophthalmologists to Differentiate Normal From Glaucomatous Eyes

Purpose: To compare the ability of Subjective assessment of optic nerve head (ONH) and retinal nerve fiber layer (RNFL) by general ophthalmologists and by a glaucoma expert with objective measurements by optical coherence tomography (Stratus OCT, Carl Zeiss Meditec Inc), confocal scanning laser ophthalmoscope (HRT III; Heidelberg Engineering, Heidelberg. Germany), and scanning laser polarimetry (GDx enhanced corneal compensation; Carl Zeiss Meditec Inc, Dublin, CA) in discriminating glaucomatous and normal eyes. Methods: Sixty-one glaucomatous and 57 normal eyes or 118 subjects Were included in the study. Three independent general ophthalmologists and I glaucoma expert evaluated ONH stereo-photographs. Receiver operating characteristic curves were constructed for each imaging technique and sensitivity at fixed specificity was estimated. Comparisons or areas under these curves (aROCs) and agreement (k) were determined between stereophoto grading and best parameter from each technique. Results: Best parameter from each technique showed larger aROC (Stratus OCT RNFL 0.92; Stratus OCT ONH vertical integrated area = 0.86; Stratus OCT macular thickness = 0.82; GDx enhanced corneal compensation = 0.91, HRT3 global cup-to-disc ratio = 0.83; HRT3 glaucoma probability score numeric area score 0.83) compared with stereophotograph grading by general ophthalmologists (0.80) in separating glaucomatous and normal eyes. Glaucoma expert stereophoto grading provided equal or larger aROC (0.92) than best parameter of each computerized imaging device. Stereophoto evaluated by a glaucoma expert showed better agreement with best parameter of each quantitative imaging technique in classifying eyes either as glaucomatous or normal compared with stereophoto grading by general ophthalmologists, The combination Of Subjective assessment of the optic disc by general ophthalmologists with RNFL objective parameters improved identification of glaucoma patients in a larger proportion than the combination of these objective parameters with Subjective assessment of the optic disc by a glaucoma expert (29.5% vs. 19.7%, respectively). Conclusions: Diagnostic ability of all imaging techniques showed better performance than subjective assessment of the ONH by general ophthalmologists, but not by It glaucoma expert, Objective RNFL measurements may provide improvement in glaucoma detection when combined with subjective assessment of the optic disc by general ophthalmologists or by a glaucoma expert.