Assessment of problem solving in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the Bologna Experts Team-Spain project and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes is described as well as some basic features regarding their reliability and validity. First conclusions on the comparison of the results achieved at two academic levels are also presented.

Assessment of learning outcomes in computing studies

The assessment of learning outcomes is a key concept in the European Credit Transfer and Accumulation System (ECTS) since credits are awarded when the assessment shows the competences which were aimed at have been developed at an appropriate level. This paper describes a study which was first part of the project of the Bologna Experts Team-Spain and then developed as an independent study. It was carried out with the overall goal to gain experience in the assessment of learning outcomes. More specifically it aimed at 1) designing procedures for the assessment of learning outcomes related to these compulsory generic competences; 2) testing some basic psychometric features that an assessment device with some consequences for the subjects being evaluated needs to prove; 3) testing different procedures of standard setting, and 4) using assessment results as orienting feedback to students and their tutors. The process of development of tests to carry out the assessment of learning outcomes related to these competences, as well as some basic features regarding their reliability and validity is described and first results on the comparison of results achieved at two academic levels, will also be described at a later stage.

Evidence of the presence of bias in subjective metrics: analysis within a family of experiments

Context: Measurement is crucial and important to empirical software engineering. Although reliability and validity are two important properties warranting consideration in measurement processes, they may be influenced by random or systematic error (bias) depending on which metric is used. Aim: Check whether, the simple subjective metrics used in empirical software engineering studies are prone to bias. Method: Comparison of the reliability of a family of empirical studies on requirements elicitation that explore the same phenomenon using different design types and objective and subjective metrics. Results: The objectively measured variables (experience and knowledge) tend to achieve more reliable results, whereas subjective metrics using Likert scales (expertise and familiarity) tend to be influenced by systematic error or bias. Conclusions: Studies that predominantly use variables measured subjectively, like opinion polls or expert opinion acquisition.

Método de Rorscharch em adultos: evidências psicométricas da Escola Francesa

O Método de Rorschach é internacionalmente utilizado e reconhecido como instrumento de avaliação psicológica em processos de investigação da personalidade. Para embasar seu adequado uso no contexto do Brasil, contínuos estudos sobre suas evidências psicométricas fazem-se necessários, sobretudo sobre seus indicadores de validade, precisão e referenciais normativos. Nesse contexto, este trabalho objetivou: a) verificar indicadores de precisão e validade do Rorschach (Escola Francesa ou de Paris), por meio da reavaliação de adultos não pacientes do estudo normativo de Pasian (1998), para checagem da estabilidade temporal (estrutural e funcional) das características de personalidade examinadas pelo método (ESTUDO 1); b) revisar e atualizar padrões normativos e atlas da Escola Francesa do Rorschach em adultos, avaliando-se eventuais especificidades de produção associadas ao sexo, à idade e à escolaridade (ESTUDO 2). No primeiro estudo foram reavaliados 88 adultos do estudo de Pasian (1998), com idade entre 34 a 69 anos, de ambos os sexos e de diferentes níveis de escolaridade (baixa, média e alta). No segundo estudo compôs-se nova amostra com 102 adultos, entre 18 e 65 anos, com sinais de desenvolvimento típico, distribuídos equitativamente em relação ao sexo e com diferentes graus de escolaridade (baixa, média e alta). Para elaboração dos dados normativos foram incluídos 66 adultos do Estudo 1, totalizando 168 casos nessa nova amostra (86 mulheres e 82 homens). Os participantes residiam no interior do Estado de São Paulo e foram avaliados, individualmente, por: a) instrumento de rastreamento de saúde mental (SRQ-20) - no estudo 1, utilizado para caracterização da amostra e no estudo 2 como critério de seleção dos participantes; b) instrumento de avaliação intelectual (Teste de Inteligência Não Verbal - INV, forma C - apenas no Estudo 2) para controle cognitivo da amostra; c) critério de classificação econômica Brasil (ABEP) e d) Método de Rorschach (Escola de Paris). Cada instrumento de avaliação psicológica foi aplicado, codificado e sistematizado conforme seus respectivos manuais técnicos. Especificamente o Método de Rorschach foi avaliado pelas diretrizes da Escola Francesa, sendo cada protocolo (de cada um dos dois estudos) examinado independentemente por dois avaliadores, chegando-se a uma classificação final dos casos. Calculou-se o índice de concordância entre examinadores pelo coeficiente Kappa para as quatro categorias de classificação das respostas do Rorschach (localização, determinante/qualidade formal, conteúdo e banalidades). Os resultados foram sistematizados inicialmente em termos descritivos (média, desvio-padrão, mediana, valor mínimo e máximo), realizando-se análises inferenciais específicas para as amostras de cada estudo em função de seus objetivos centrais. No Estudo 1, os achados relativos aos índices de correlação entre as duas avaliações dos 88 voluntários variaram entre 0,72 a -0,005, evidenciando estabilidade em um conjunto das características de personalidade dos adultos examinados após 15 anos, configurando evidências empíricas de precisão e de validade de método projetivo. Houve variáveis do Rorschach que não apresentaram resultados estáveis entre os dois momentos avaliativos (fórmulas vivenciais), sugerindo se tratarem de indicadores técnicos relativos ao funcionamento da personalidade, mais do que componentes estruturais. As análises do Estudo 2 apontaram reduzida influência dos fatores relacionados ao sexo, à escolaridade e à idade sobre as variáveis do Rorschach, não indicando a necessidade de normas específicas para grupos em termos desses fatores. Por fim, foi elaborado novo atlas de referência do Método de Rorschach (Escola de Paris) no contexto brasileiro, seguindo-se as diretrizes técnicocientíficas nacionais e internacionais da área. Os dados fortalecem a relevância dos estudos de natureza psicométrica para embasar adequadas análises interpretativas desse instrumento de avaliação psicológica (FAPESP e CAPES/PDSE).

Psychometric Investigation of the Check-In, Check-Out Fidelity of Implementation Measure

Suspension and expulsion are utilized frequently and disproportionality in schools in the United States. Many schools utilize Positive Behavioral Interventions and Supports (PBIS), a tiered framework to prevent problem behavior and reduce the use of discipline practices (Sugai et al., 2000). Check-In, Check-Out (CICO) is a targeted group behavioral intervention that is utilized within this framework in schools to prevent severe problem behavior in students that are beginning to exhibit externalizing and/or internalizing behavioral needs; thus, preventing the use of exclusionary discipline practices (Crone et al., 2010; Hawken & Horner, 2003). As the use of CICO in schools continues to grow, so too does the need for an instrument measuring its fidelity of implementation. The purpose of this study was to investigate the reliability and validity of the Check-In, Check-Out Fidelity of Implementation Measure (Crone et al., 2010), an instrument created to measure the fidelity of implementation of CICO intervention. This study assessed the psychometric properties of the instrument utilizing an archival data set collected by the statewide PBIS initiative in a western state in the U.S. The results demonstrated promising content validity, construct validity, internal consistency, and interrater reliability. A unidimensional structure was determined to be the best structure for the instrument based on parsimony and the strong results obtained from the item loadings, internal consistency, and interrater reliability. Implications for use and future research are discussed.

Metodología para el estudio del brand-placement con enfoque de género: más allá del análisis de contenido

La mayoría de los estudios realizados sobre el tratamiento de género en la publicidad audiovisual se ciñen al formato convencional. Los nuevos formatos no se adecuan siempre a estos patrones. Con este fin, se ha elaborado un protocolo de análisis para estudiar la estructura del brand-placement, como forma no convencional de publicidad, en temas de género en la ficción española. Esta herramienta ha sido sometida a rigurosos controles de fiabilidad y validez con el ánimo de garantizar la confiabilidad y rigurosidad científica de las mismas, tanto en su composición como en su aplicación.

Fiabilidad y validez de la versión española de una escala de autoeficacia en la lactancia materna

Objetivo: El objetivo de este trabajo es validar lingüísticamente la Breastfeeding Self-Efficacy Scale-Short Form (BSES-SF) al español y determinar sus características psicométricas. Personas y método: Estudio instrumental que se llevó a cabo en dos hospitales de la provincia de Alicante. Tras los procedimientos de traducción y retrotraducción, una muestra accidental de 150 madres lactantes cumplimentaron, a las 48 horas del parto y aún hospitalizadas, la versión española de la BSES-SF y un cuestionario con variables sociodemográficas, obstétricas y sobre el estatus de lactancia materna (LM) al alta. A las 6 semanas posparto se obtuvieron, nuevamente, datos sobre el estatus de LM mediante una encuesta telefónica. Resultados: El coeficiente alfa de Cronbach fue de 0,79. Las mujeres con experiencia previa y con experiencia previa muy positiva en LM, con puntuaciones más altas en el ítem global de autoeficacia y con más hijos, tuvieron mayores puntuaciones en la versión española de la BSES-SF. Dos de los ítems (9 y 10) de la versión española presentaron correlaciones ítems-test corregidas <0,30, y saturaron por debajo de 0,30 al forzar la extracción a un factor en el análisis de componentes principales. Las puntuaciones de la escala al alta hospitalaria no guardaron relación con el estatus de LM a las 6 semanas posparto. Conclusiones: El instrumento muestra aceptables evidencias de fiabilidad y validez, aunque dos de los ítems deberían ser revisados.

Psychometric properties of the School Anxiety Inventory-Short Version in Spanish secondary education students

Background: The School Anxiety Inventory (SAI) can be applied in different fields of psychology. However, due to the inventory’s administration time, it may not be useful in certain situations. To address this concern, the present study developed a short version of the SAI (the SAI-SV). Method: This study examined the reliability and validity evidence drawn from the scores of the School Anxiety Inventory-Short Version (SAI-SV) using a sample of 2,367 (47.91% boys) Spanish secondary school students, ranging from 12 to 18 years of age. To analyze the dimensional structure of the SAI-SV, exploratory and confirmatory factor analyses were applied. Internal consistency and test-retest reliability were calculated for SAISV scores. Results: A correlated three-factor structure related to school situations (Anxiety about Aggression, Anxiety about Social Evaluation, and Anxiety about Academic Failure) and a three-factor structure related to the response systems of anxiety (Physiological Anxiety, Cognitive Anxiety, and Behavioral Anxiety) were identified and supported. The internal consistency and test-retest reliability were determined to be appropriate. Conclusions: The reliability and validity evidence based on the internal structure of SAI-SV scores was satisfactory.

Evaluación del bienestar espiritual en pacientes en cuidados paliativos

Objetivo: La evaluación del bienestar espiritual del paciente es un aspecto crítico y fundamental en la atención holística y multidisciplinar. Disponer de un instrumento válido de evaluación de la espiritualidad con finalidad tanto investigadora como clínica es muy conveniente. En este estudio se examina la presencia de bienestar espiritual, o sentido de la vida, en pacientes en cuidados paliativos a través de un cuestionario con garantías psicométricas de calidad. Pacientes y Métodos: Han participado en este estudio con diseño transversal un total de 60 pacientes en cuidados paliativos. El instrumento básico empleado ha sido la versión española del Cuestionario del Sentido de la Vida (Meaning in Life Scale), de 21 ítems y 4 escalas: Propósito, Falta de significado, Paz, y Beneficios de la espiritualidad. Se proporciona, también, una puntuación global de espiritualidad. Se registraron, además, variables de tipo clínico y sociodemográfico, así como estimaciones del estado de salud, calidad de vida (general y actual), felicidad personal, el grado de religiosidad y la creencia en la existencia de vida ultraterrena. Resultados: Los resultados muestran que la versión española de este instrumento (Cuestionario del Sentido de la Vida [Meaning in Life Scale]) es una medida de bienestar espiritual con garantías psicométricas de calidad (buena fiabilidad y validez), adecuada para evaluar la complejas exigencias generadas por la enfermedad crónica del paciente en cuidados paliativos. El bienestar espiritual se relaciona significativamente con diversas variables de calidad de vida, percepción de salud, felicidad personal o religiosidad. No existe una relación significativa entre las puntuaciones de espiritualidad y variables sociodemográficas como la edad o el sexo. La presencia de bienestar espiritual en estos pacientes es más baja de la esperada. Conclusión: Este cuestionario de espiritualidad resulta un instrumento válido para valorar las 4 dimensiones básicas del bienestar espiritual. Los resultados sugieren que considerar y evaluar el bienestar espiritual de los pacientes en cuidados paliativos puede ser de ayuda para la práctica clínica. La presencia de espiritualidad parece desempeñar un papel importante en el bienestar psicológico, el estado de salud y la calidad de vida percibidos por el paciente en cuidados paliativos. El grado de bienestar espiritual de estos pacientes es relativamente bajo.

Validation of a Spanish Version of the Lille Apathy Rating Scale for Parkinson’s Disease

Introduction. To date, no rating scales for detecting apathy in Parkinson’s disease (PD) patients have been validated in Spanish. For this reason, the aim of this study was to validate a Spanish version of Lille apathy rating scale (LARS) in a cohort of PD patients from Spain. Participants and Methods. 130 PD patients and 70 healthy controls were recruited to participate in the study. Apathy was measured using the Spanish version of LARS and the neuropsychiatric inventory (NPI). Reliability (internal consistency, test-retest, and interrater reliability) and validity (construct, content, and criterion validity) were measured. Results. Interrater reliability was 0.93. Cronbach’s α for LARS was 0.81. The test-retest correlation coefficient was 0.97. The correlation between LARS and NPI scores was 0.61. The optimal cutoff point under the ROC curve was , whereas the value derived from healthy controls was . The prevalence of apathy in our population tested by LARS was 42%. Conclusions. The Spanish version of LARS is a reliable and useful tool for diagnosing apathy in PD patients. Total LARS score is influenced by the presence of depression and cognitive impairment. However, both disorders are independent identities with respect to apathy. The satisfactory reliability and validity of the scale make it an appropriate instrument for screening and diagnosing apathy in clinical practice or for research purposes.

Validation of a Spanish version of the Distance Education Learning Environments Survey (DELES) in Spain

The aim of this study was to examine the validity of the Spanish version of the Distance Education Learning Environments Survey (Sp-DELES). This instrument assesses students’ perceptions of virtual learning environments using six scales: Instructor Support, Student Interaction and Collaboration, Personal Relevance, Authentic Learning, Active Learning, and Autonomy. Further, the Sp-DELES includes an additional scale that assesses students’ Satisfaction with their classes. The original DELES has been used in at least 27 independent studies with strong reliability and validity. For this study, we sampled 265 students from the University of Alicante enrolled in various hybrid and distance education courses taught by the Department of Health Psychology. We analysed the Sp-DELES for validity using principal component factor analysis with varimax rotation, and for reliability using Cronbach’s alpha. The Sp-DELES exhibited good reliability (Cronbach’s alpha for the scales ranging from 0.86 to 0.97) and the original six-factor structure was replicated and accounted for 72.9 % of the total variance. Overall the results are consistent with those of the original English-language version of the instrument. The Sp-DELES has proven to be a reliable and valid instrument for assessing psychosocial learning environments in tertiary-level hybrid and distance-education settings.

Validation of the Spanish version of mackey childbirth satisfaction rating scale

Background: The “Mackey Childbirth Satisfaction Rating Scale” (MCSRS) is a complete non-validated scale which includes the most important factors associated with maternal satisfaction. Our primary purpose was to describe the internal structure of the scale and validate the reliability and validity of concept of its Spanish version MCSRS-E. Methods: The MCSRS was translated into Spanish, back-translated and adapted to the Spanish population. It was then administered following a pilot test with women who met the study participant requirements. The scale structure was obtained by performing an exploratory factorial analysis using a sample of 304 women. The structures obtained were tested by conducting a confirmatory factorial analysis using a sample of 159 women. To test the validity of concept, the structure factors were correlated with expectations prior to childbirth experiences. McDonald’s omegas were calculated for each model to establish the reliability of each factor. The study was carried out at four University Hospitals; Alicante, Elche, Torrevieja and Vinalopo Salud of Elche. The inclusion criteria were women aged 18–45 years old who had just delivered a singleton live baby at 38–42 weeks through vaginal delivery. Women who had difficulty speaking and understanding Spanish were excluded. Results: The process generated 5 different possible internal structures in a nested model more consistent with the theory than other internal structures of the MCSRS applied hitherto. All of them had good levels of validation and reliability. Conclusions: This nested model to explain internal structure of MCSRS-E can accommodate different clinical practice scenarios better than the other structures applied to date, and it is a flexible tool which can be used to identify the aspects that should be changed to improve maternal satisfaction and hence maternal health.

Utility of the Social Responsiveness Scale- Parent Report (SRS-Parent) as a Diagnostic Tool for Autism Identification

Thesis (Ph.D.)--University of Washington, 2016-06

WHOQOL-BREF as a measure of quality of life in older patients with depression

Objective: To assess the reliability and validity of a brief measure of quality of life recently developed by the World Health Organization, the WHOQOL-BREF, and to examine its association with a variety of clinical and sociodemographic factors in older depressed patients. Design: Cross-sectional study. Methods: Older depressed patients (N=41) underwent diagnostic assessment using the Composite International Diagnostic Interview (CIDI) and were independently assessed on a variety of measures including the WHOQOL-BREF (a 26-item self-report questionnaire generating four domain scores), Hamilton Depression Rating Scale (HAM-D); Geriatric Depression Scale (GDS); Mini-mental State Examination (MMSE); Modified Barthel Index (MBI); Instrumental activities of daily living (IADL), and measures of physical health status and social relationships. Estimates of inter-rater and test-retest reliability, and concurrent validity were made. Results: 39 subjects completed the study. The majority of subjects (94.9%) received a diagnosis of DSM-IV Major Depressive Disorder. Levels of comorbidity were high. Three of the four domains of the WHOQOL-BREF (Physical, Psychological and Environment domains) demonstrated satisfactory reliability and validity. However, the Social Relationships domain exhibited poor validity. Quality of life scores were strongly correlated with severity of depression, number of self-reported physical symptoms and self-assessed general health status. There was no relationship between diagnostic comorbidity and quality of life scores. Conclusions: The WHOQOL-BREF was successfully administered to older depressed patients although the concurrent validity of one of its four domains was poor. Quality of life scores were strongly correlated with severity of depression, raising the issue of measurement redundancy.

California older adult Stroop test (COAST): Development of a Stroop test adapted for geriatric populations

An adaptation of the traditional Stroop test, the California Older Adult Stroop Test (COAST) (Pachana, Marcopulos, Yoash-Gantz & Thompson, 1995), has been developed specifically for use with a geriatric population, utilizing larger typeface, fewer items (50) per task, and more easily distinguished colors (red, yellow and green). Test-retest reliability and validity data are reviewed for both control and clinical populations. Increased error rates on the Stroop test compared to the COAST were found for the color and color/word interference tasks. These results are discussed in terms of changes in the visual system with increasing age. The implications for better test sensitivity with the COAST for older adult populations are discussed.