How to assess the external validity of therapeutic trials: a conceptual approach

Background External validity of study results is an important issue from a clinical point of view. From a methodological point of view, however, the concept of external validity is more complex than it seems to be at first glance. Methods Methodological review to address the concept of external validity. Results External validity refers to the question whether results are generalizable to persons other than the population in the original study. The only formal way to establish the external validity would be to repeat the study for that specific target population. We propose a three-way approach for assessing the external validity for specified target populations. (i) The study population might not be representative for the eligibility criteria that were intended. It should be addressed whether the study population differs from the intended source population with respect to characteristics that influence outcome. (ii) The target population will, by definition, differ from the study population with respect to geographical, temporal and ethnical conditions. Pondering external validity means asking the question whether these differences may influence study results. (iii) It should be assessed whether the study's conclusions can be generalized to target populations that do not meet all the eligibility criteria. Conclusion Judging the external validity of study results cannot be done by applying given eligibility criteria to a single target population. Rather, it is a complex reflection in which prior knowledge, statistical considerations, biological plausibility and eligibility criteria all have place.

Comparison of objectively measured motor behavior with ratings of the motor behavior domain of the Bern Psychopathology Scale (BPS) in schizophrenia

Motor symptoms in schizophrenia occur frequently and are relevant to diagnosis and antipsychotic therapy. To date motor symptoms are difficult to assess and their pathobiology is a widely unresolved issue. The Bern Psychopathology Scale for the assessment of system-specific psychotic symptoms (BPS) was designed to identify homogenous patient groups by focusing on three domains: language, affectivity and motor behavior. The present study aimed to validate the motor behavior domain of the BPS using wrist actigraphy. In total, 106 patients were rated with the BPS and underwent 24 h continuous actigraphy recording. The ratings of the global severity of the motor behavior domain (GSM) as well as the quantitative and the subjective items of the motor behavior domain of the BPS were significantly associated with actigraphic variables. In contrast, the qualitative items of the motor domain failed to show an association with actigraphy. Likewise, scores of the language and the affectivity domains were not related to actigraphic measures. In conclusion, we provided substantial external validity for global, quantitative and subjective ratings of the BPS motor behavior domain. Thus, the BPS is suitable to assess the dimension of quantitative motor behavior in the schizophrenia spectrum.

Physiological correlates of aesthetic perception of artworks in a museum

Arts experts are commonly skeptical of applying scientific methods to aesthetic experiencing, which remains a field of study predominantly for the humanities. Laboratory research has however indicated that artworks may elicit emotional and physiological responses. Yet, this line of aesthetics research has previously suffered from insufficient external validity. We therefore conducted a study in which aesthetic perception was monitored in a fine-art museum, unrestricting to the viewers’ freedom of aesthetic choice. Visitors were invited to wear electronic gloves through which their locomotion, heart rate and skin conductance were continuously recorded. Emotional and aesthetic responses to selected works of an exhibition were assessed using a customized questionnaire. In a sample of 373 adult participants, we found that physiological responses during perception of an artwork were significantly related to aesthetic-emotional experiencing. The dimensions ‘Aesthetic Quality’, ‘Surprise/Humor’, ‘Dominance’ and ‘Curatorial Quality’ were associated with cardiac measures (heart rate variability, heart rate level) and skin conductance variability. This is first evidence that aesthetics can be statistically grounded in viewers’ physiology in an ecologically valid environment, the art gallery, enhancing our understanding of the effects of artworks and their curatorial staging.

The rationale for a spine registry

In the discussion about the rationale for spine registries, two basic questions have to be answered. The first one deals with the value of orthopaedic registries per se, considering them as observational studies and comparing the evidence they generate with that of randomised controlled trials. The second question asks if the need for registries in spine surgery is similar to that in the arthroplasty sector. The widely held view that randomised controlled trials are the 'gold standard' for evaluation and that observational methods have little or no value ignores the limitations of randomised trials. They may prove unnecessary, inappropriate, impossible, or inadequate. In addition, the external validity and hence the ability to make generalisations about the results of randomised trials is often low. Therefore, the false conflict between those who advocate randomised trials in all situations and those who believe observational data provide sufficient evidence needs to be replaced with mutual recognition of their complementary roles. The fact that many surgical techniques or technologies were introduced into the field of spine surgery without randomised trials or prospective cohort comparisons makes obvious an even increased need for spine registries compared to joint arthroplasty. An essential methodological prerequisite for a registry is a common terminology for reporting results and a sophisticated technology that networks all participants so that one central data pool is created and accessed. Recognising this need, the Spine Society of Europe has researched and developed Spine Tango, the first European spine registry, which can be accessed under www.eurospine.org.

Back related outcome assessment instruments

A literature review of the most widely used condition specific, self administered assessment questionnaires for low back pain had been undertaken. General and historic aspects, reliability, responsiveness and minimum clinically important difference, external validity, floor and ceiling effects, and available languages were analysed. These criteria, however, are only part of the consideration. Of similar importance are the content, wording of questions and answers in each of the six questionnaires and an analysis of the different score results. The issue of score bias is discussed and suggestions are given in order to increase the construct validity in the practical use of the individual questionnaires.

Effectiveness, safety and cost-effectiveness of homeopathy in general practice - summarized health technology assessment

Introduction: The Health Technology Assessment report on effectiveness, cost-effectiveness and appropriateness of homeopathy was compiled on behalf of the Swiss Federal Office for Public Health (BAG) within the framework of the 'Program of Evaluation of Complementary Medicine (PEK)'. Materials and Methods: Databases accessible by Internet were systematically searched, complemented by manual search and contacts with experts, and evaluated according to internal and external validity criteria. Results: Many high-quality investigations of pre-clinical basic research proved homeopathic high-potencies inducing regulative and specific changes in cells or living organisms. 20 of 22 systematic reviews detected at least a trend in favor of homeopathy. In our estimation 5 studies yielded results indicating clear evidence for homeopathic therapy. The evaluation of 29 studies in the domain 'Upper Respiratory Tract Infections/Allergic Reactions' showed a positive overall result in favor of homeopathy. 6 out of 7 controlled studies were at least equivalent to conventional medical interventions. 8 out of 16 placebocontrolled studies were significant in favor of homeopathy. Swiss regulations grant a high degree of safety due to product and training requirements for homeopathic physicians. Applied properly, classical homeopathy has few side-effects and the use of high-potencies is free of toxic effects. A general health-economic statement about homeopathy cannot be made from the available data. Conclusion: Taking internal and external validity criteria into account, effectiveness of homeopathy can be supported by clinical evidence and professional and adequate application be regarded as safe. Reliable statements of cost-effectiveness are not available at the moment. External and model validity will have to be taken more strongly into consideration in future studies.

Traditional chinese medicine (phytotherapy): health technology assessment report - selected aspects

Objective: A summary of main aspects from a Health Technology Assessment report on Traditional Chinese Medicine (TCM) in Switzerland concerning effectiveness and safety is given. Materials and Methods: Literature search was performed through 13 databases, by scanning reference lists of articles and by contacting experts. Assessed were quality of documentation, internal and external validity. Results: Effectiveness: 43 articles concerning 'gastrointestinal tract and liver' were assessed. The studies covering 7,436 patients were undertaken in China (35), Japan (3), USA (2) and Australia (3); 33/43 being controlled studies. 34/40 show significantly better results in the TCM-treated group. A comparison of studies on results of treatment based on a diagnosis according to TCM criteria and studies on results of treatment according to Western diagnosis shows that treatment based on TCM diagnosis improves the result. The comparison of treatment by individual medication and standard medication showed a trend in favor of individual medication. Safety: TCM training and practice for physicians in Switzerland are officially regulated. Side effects occur, but no severe effects have been registered up to now in Switzerland. TCM medicinals are imported; admission regulations are being installed. Problems due to production abroad, Internet trade, self-medication or admixtures are possible. Conclusion: The evaluation of the literature search provides evidence for a basic clinical effectiveness of TCM therapy. Severe side effects were not observed in Switzerland. Regulations for trading and use of medicinals prevent treatment risks. Further clinical studies in a Western context are required.

Prescription procedures in medication for relapse prevention after inpatient treatment for alcohol use disorders in Switzerland

In randomized controlled trials with high internal validity, pharmacotherapy using acamprosate, naltrexone, and, to a somewhat lesser extent, disulfiram has proved effective in preventing relapse in patients with alcohol use disorders (AUD). There remains, however, a paucity of studies with sufficient external validity in which the effectiveness of pharmacotherapy in clinical practice is investigated. This study aimed to make a contribution to close this gap in research.

Methods and representativeness of a European survey in children and adolescents: the KIDSCREEN study

BACKGROUND: The objective of the present study was to compare three different sampling and questionnaire administration methods used in the international KIDSCREEN study in terms of participation, response rates, and external validity. METHODS: Children and adolescents aged 8-18 years were surveyed in 13 European countries using either telephone sampling and mail administration, random sampling of school listings followed by classroom or mail administration, or multistage random sampling of communities and households with self-administration of the survey materials at home. Cooperation, completion, and response rates were compared across countries and survey methods. Data on non-respondents was collected in 8 countries. The population fraction (PF, respondents in each sex-age, or educational level category, divided by the population in the same category from Eurostat census data) and population fraction ratio (PFR, ratio of PF) and their corresponding 95% confidence intervals were used to analyze differences by country between the KIDSCREEN samples and a reference Eurostat population. RESULTS: Response rates by country ranged from 18.9% to 91.2%. Response rates were highest in the school-based surveys (69.0%-91.2%). Sample proportions by age and gender were similar to the reference Eurostat population in most countries, although boys and adolescents were slightly underrepresented (PFR <1). Parents in lower educational categories were less likely to participate (PFR <1 in 5 countries). Parents in higher educational categories were overrepresented when the school and household sampling strategies were used (PFR = 1.78-2.97). CONCLUSION: School-based sampling achieved the highest overall response rates but also produced slightly more biased samples than the other methods. The results suggest that the samples were sufficiently representative to provide reference population values for the KIDSCREEN instrument.

Measuring finger joint cartilage by ultrasound as a promising alternative to conventional radiograph imaging

OBJECTIVE: To evaluate the reliability and validity of a novel ultrasound (US) imaging method to measure metacarpophalangeal (MCP) and proximal interphalangeal (PIP) finger joint cartilage. METHODS: We examined 48 patients with rheumatoid arthritis (RA), 18 patients with osteoarthritis (OA), 24 patients with unclassified arthritis of the finger joints, and 34 healthy volunteers. The proximal cartilage layer of MCP and PIP joints for fingers 2-5 was bilaterally visualized from a posterior view, with joints in approximately 90 degrees flexion. Cartilage thickness was measured with integrated tools on static images. External validity was assessed by measuring radiologic joint space width (JSW) and a numeric joint space narrowing (JSN) score in patients with RA. RESULTS: Precise measurement was possible for 97.5% of MCP and 94.2% of PIP joints. Intraclass correlation coefficients for bilateral total joint US scores were 0.844 (95% confidence interval [95% CI] 0.648-0.935) for interobserver comparisons and 0.928 (95% CI 0.826-0.971) for intraobserver comparisons (using different US devices). The US score correlated with JSN for both hands (adjusted R(2) = 0.513, P < 0.001) and JSW of the same finger joints (adjusted R(2) = 0.635, P < 0.001). Reduced cartilage shown by US allowed discrimination of early symptomatic OA versus early RA and healthy joints. In patients with RA, US scores correlated with duration of treatment-resistant, progressive RA. CONCLUSION: The US method of direct visualization and quantification of cartilage in MCP and PIP joints is objective, reliable, valid, and can be useful for diagnostic purposes in patients with arthritis.

Image and imaging an emergency department: expense and benefit of different quality assessment methods

Introduction. In this era of high-tech medicine, it is becoming increasingly important to assess patient satisfaction. There are several methods to do so, but these differ greatly in terms of cost, time, and labour and external validity. The aim of this study is to describe and compare the structure and implementation of different methods to assess the satisfaction of patients in an emergency department. Methods. The structure and implementation of the different methods to assess patient satisfaction were evaluated on the basis of a 90-minute standardised interview. Results. We identified a total of six different methods in six different hospitals. The average number of patients assessed was 5012, with a range from 230 (M5) to 20 000 patients (M2). In four methods (M1, M3, M5, and M6), the questionnaire was composed by a specialised external institute. In two methods, the questionnaire was created by the hospital itself (M2, M4).The median response rate was 58.4% (range 9-97.8%). With a reminder, the response rate increased by 60% (M3). Conclusion. The ideal method to assess patient satisfaction in the emergency department setting is to use a patient-based, in-emergency department-based assessment of patient satisfaction, planned and guided by expert personnel.