Development of test items for the entry-level social worker class in selected State and local jurisdictions /

Photocopy. Springfield, Va. : U.S. Dept. of Commerce, National Technical Information Service, 1977. -- v, 74 leaves ; 28 cm.

Applying a new version of the Brazilian-Portuguese UPSIT smell test in Brazil

Standardized olfactory tests are now available to quantitatively assess disorders of olfaction. A Brazilian-Portuguese version of the University of Pennsylvania Smell Identification Test (UPSIT) is currently being developed specifically for the Brazilian population. The most recent Brazilian-Portuguese version of the UPSIT (UPSIT-Br2) was administered to 88 Brazilian subjects who had no history of neurological or otorhinolaryngological disease. UPSIT-Br2 scores decreased with age, were lower in men than in women, and were lower in subjects with lower income. The degree to which the poorer performance of subjects with lower socio-economic status reflects lack of familiarity with test items is not known. Although this version of the UPSIT provides a sensitive and useful test of smell function for the Brazilian population, a revision of some test items is needed to achieve comparable norms to those found using the North American UPSIT in the United States.

The position effect in a Rasch-homogenous test: A fixed-links modeling approach.

The position effect describes the influence of just-completed items in a psychological scale on subsequent items. This effect has been repeatedly reported for psychometric reasoning scales and is assumed to reflect implicit learning during testing. One way to identify the position effect is fixed-links modeling. With this approach, two latent variables are derived from the test items. Factor loadings of one latent variable are fixed to 1 for all items to represent ability-related variance. Factor loadings on the second latent variable increase from the first to the last item describing the position effect. Previous studies using fixed-links modeling on the position effect investigated reasoning scales constructed in accordance with classical test theory (e.g., Raven’s Progressive Matrices) but, to the best of our knowledge, no Rasch-scaled tests. These tests, however, meet stronger requirements on item homogeneity. In the present study, therefore, we will analyze data from 239 participants who have completed the Rasch-scaled Viennese Matrices Test (VMT). Applying a fixed-links modeling approach, we will test whether a position effect can be depicted as a latent variable and separated from a latent variable representing basic reasoning ability. The results have implications for the assumption of homogeneity in Rasch-homogeneous tests.

Automatized Generation of Metadata and Calibrated Test Item Banks for E-learning

Report published in the Proceedings of the National Conference on "Education in the Information Society", Plovdiv, May, 2013

Cognitive-didactic Model for Making Testing Items in Physics

Report published in the Proceedings of the National Conference on "Education and Research in the Information Society", Plovdiv, May, 2015

Student, school, and country differences in sustained test-taking effort in the 2009 PISA reading assessment

In this article, the change in examinee effort during an assessment, which we will refer to as persistence, is modeled as an effect of item position. A multilevel extension is proposed to analyze hierarchically structured data and decompose the individual differences in persistence. Data from the 2009 Program of International Student Achievement (PISA) reading assessment from N = 467,819 students from 65 countries are analyzed with the proposed model, and the results are compared across countries. A decrease in examinee effort during the PISA reading assessment was found consistently across countries, with individual differences within and between schools. Both the decrease and the individual differences are more pronounced in lower performing countries. Within schools, persistence is slightly negatively correlated with reading ability; but at the school level, this correlation is positive in most countries. The results of our analyses indicate that it is important to model and control examinee effort in low-stakes assessments. (DIPF/Orig.)

Developing multiple - choice questions in mathematics

In this paper we wish to illustrate different perspectives used to create Multiple-Choice questions and we will show how we can improve these in the construction of math tests. As it is known, web technologies have a great influence on student’s behaviour. Based on an on-line project beginning at 2007 which has been contributing to help students on their individual work, we would like to share our experience and thoughts with colleagues who have a common concern when they have the task of constructing Multiple-Choice tests. We feel that Multiple-Choice tests play an important and a very useful supporting role in selfevaluation or self-examination of our students. Nonetheless, good Multiple–Choice Test Items are generally more complex and time-consuming to create than other types of tests. It requires a certain amount of skill. However, this skill maybe increases through study, practice and experience. This paper discusses a number of issues related to the use of Multiple-Choice questions, lists the advantages and disadvantages of this question format contrasting it with open questions. Some examples are given in this context.

Robustesse du modèle de Rasch unidimensionnel à la violation de l’hypothèse d’unidimensionnalité.

Le logiciel de simulation des données et d'analyse est Conquest V.3

Infants Can Learn Decontextualized Words Before Their First Birthday

Can infants below age 1 year learn words in one context and understand them in another? To investigate this question, two groups of parents trained infants from age 9 months on 8 categories of common objects. A control group received no training. At 12 months, infants in the experimental groups, but not in the control group, showed comprehension of the words in a new context. It appears that infants under 1 year old can learn words in a decontextualized, as distinct from a context-bound, fashion. Perceptual variability within the to-be-learned categories, and the perceptual similarity between training sets and the novel test items, did not appear to affect this learning.

A web-based system for visualizing upper limb motor performance of Parkinson’s disease patients

Objective To design, develop and set up a web-based system for enabling graphical visualization of upper limb motor performance (ULMP) of Parkinson’s disease (PD) patients to clinicians. Background Sixty-five patients diagnosed with advanced PD have used a test battery, implemented in a touch-screen handheld computer, in their home environment settings over the course of a 3-year clinical study. The test items consisted of objective measures of ULMP through a set of upper limb motor tests (finger to tapping and spiral drawings). For the tapping tests, patients were asked to perform alternate tapping of two buttons as fast and accurate as possible, first using the right hand and then the left hand. The test duration was 20 seconds. For the spiral drawing test, patients traced a pre-drawn Archimedes spiral using the dominant hand, and the test was repeated 3 times per test occasion. In total, the study database consisted of symptom assessments during 10079 test occasions. Methods Visualization of ULMP The web-based system is used by two neurologists for assessing the performance of PD patients during motor tests collected over the course of the said study. The system employs animations, scatter plots and time series graphs to visualize the ULMP of patients to the neurologists. The performance during spiral tests is depicted by animating the three spiral drawings, allowing the neurologists to observe real-time accelerations or hesitations and sharp changes during the actual drawing process. The tapping performance is visualized by displaying different types of graphs. Information presented included distribution of taps over the two buttons, horizontal tap distance vs. time, vertical tap distance vs. time, and tapping reaction time over the test length. Assessments Different scales are utilized by the neurologists to assess the observed impairments. For the spiral drawing performance, the neurologists rated firstly the ‘impairment’ using a 0 (no impairment) – 10 (extremely severe) scale, secondly three kinematic properties: ‘drawing speed’, ‘irregularity’ and ‘hesitation’ using a 0 (normal) – 4 (extremely severe) scale, and thirdly the probable ‘cause’ for the said impairment using 3 choices including Tremor, Bradykinesia/Rigidity and Dyskinesia. For the tapping performance, a 0 (normal) – 4 (extremely severe) scale is used for first rating four tapping properties: ‘tapping speed’, ‘accuracy’, ‘fatigue’, ‘arrhythmia’, and then the ‘global tapping severity’ (GTS). To achieve a common basis for assessment, initially one neurologist (DN) performed preliminary ratings by browsing through the database to collect and rate at least 20 samples of each GTS level and at least 33 samples of each ‘cause’ category. These preliminary ratings were then observed by the two neurologists (DN and PG) to be used as templates for rating of tests afterwards. In another track, the system randomly selected one test occasion per patient and visualized its items, that is tapping and spiral drawings, to the two neurologists. Statistical methods Inter-rater agreements were assessed using weighted Kappa coefficient. The internal consistency of properties of tapping and spiral drawing tests were assessed using Cronbach’s α test. One-way ANOVA test followed by Tukey multiple comparisons test was used to test if mean scores of properties of tapping and spiral drawing tests were different among GTS and ‘cause’ categories, respectively. Results When rating tapping graphs, inter-rater agreements (Kappa) were as follows: GTS (0.61), ‘tapping speed’ (0.89), ‘accuracy’ (0.66), ‘fatigue’ (0.57) and ‘arrhythmia’ (0.33). The poor inter-rater agreement when assessing “arrhythmia” may be as a result of observation of different things in the graphs, among the two raters. When rating animated spirals, both raters had very good agreement when assessing severity of spiral drawings, that is, ‘impairment’ (0.85) and irregularity (0.72). However, there were poor agreements between the two raters when assessing ‘cause’ (0.38) and time-information properties like ‘drawing speed’ (0.25) and ‘hesitation’ (0.21). Tapping properties, that is ‘tapping speed’, ‘accuracy’, ‘fatigue’ and ‘arrhythmia’ had satisfactory internal consistency with a Cronbach’s α coefficient of 0.77. In general, the trends of mean scores of tapping properties worsened with increasing levels of GTS. The mean scores of the four properties were significantly different to each other, only at different levels. In contrast from tapping properties, kinematic properties of spirals, that is ‘drawing speed’, ‘irregularity’ and ‘hesitation’ had a questionable consistency among them with a coefficient of 0.66. Bradykinetic spirals were associated with more impaired speed (mean = 83.7 % worse, P < 0.001) and hesitation (mean = 77.8% worse, P < 0.001), compared to dyskinetic spirals. Both these ‘cause’ categories had similar mean scores of ‘impairment’ and ‘irregularity’. Conclusions In contrast from current approaches used in clinical setting for the assessment of PD symptoms, this system enables clinicians to animate easily and realistically the ULMP of patients who at the same time are at their homes. Dynamic access of visualized motor tests may also be useful when observing and evaluating therapy-related complications such as under- and over-medications. In future, we foresee to utilize these manual ratings for developing and validating computer methods for automating the process of assessing ULMP of PD patients.

PokerMapper: : mapping executive functions, poker playing ability and responsible gambling in online environments

Research objectives Poker and responsible gambling both entail the use of the executive functions (EF), which are higher-level cognitive abilities. The main objective of this work was to assess if online poker players of different ability show different performances in their EF and if so, which functions are the most discriminating ones. The secondary objective was to assess if the EF performance can predict the quality of gambling, according to the Gambling Related Cognition Scale (GRCS), the South Oaks Gambling Screen (SOGS) and the Problem Gambling Severity Index (PGSI). Sample and methods The study design consisted of two stages: 46 Italian active players (41m, 5f; age 32±7,1ys; education 14,8±3ys) fulfilled the PGSI in a secure IT web system and uploaded their own hand history files, which were anonymized and then evaluated by two poker experts. 36 of these players (31m, 5f; age 33±7,3ys; education 15±3ys) accepted to take part in the second stage: the administration of an extensive neuropsychological test battery by a blinded trained professional. To answer the main research question we collected all final and intermediate scores of the EF tests on each player together with the scoring on the playing ability. To answer the secondary research question, we referred to GRCS, PGSI and SOGS scores.  We determined which variables that are good predictors of the playing ability score using statistical techniques able to deal with many regressors and few observations (LASSO, best subset algorithms and CART). In this context information criteria and cross-validation errors play a key role for the selection of the relevant regressors, while significance testing and goodness-of-fit measures can lead to wrong conclusions.   Preliminary findings We found significant predictors of the poker ability score in various tests. In particular, there are good predictors 1) in some Wisconsin Card Sorting Test items that measure flexibility in choosing strategy of problem-solving, strategic planning, modulating impulsive responding, goal setting and self-monitoring, 2) in those Cognitive Estimates Test variables related to deductive reasoning, problem solving, development of an appropriate strategy and self-monitoring, 3) in the Emotional Quotient Inventory Short (EQ-i:S) Stress Management score, composed by the Stress Tolerance and Impulse Control scores, and in the Interpersonal score (Empathy, Social Responsibility, Interpersonal Relationship). As for the quality of gambling, some EQ-i:S scales scores provide the best predictors: General Mood for the PGSI; Intrapersonal (Self-Regard; Emotional Self-Awareness, Assertiveness, Independence, Self-Actualization) and Adaptability  (Reality Testing, Flexibility, Problem Solving) for the SOGS, Adaptability for the GRCS. Implications for the field Through PokerMapper we gathered knowledge and evaluated the feasibility of the construction of short tasks/card games in online poker environments for profiling users’ executive functions. These card games will be part of an IT system able to dynamically profile EF and provide players with a feedback on their expected performance and ability to gamble responsibly in that particular moment. The implementation of such system in existing gambling platforms could lead to an effective proactive tool for supporting responsible gambling. 

A avaliação da compreensão leitora em espanhol língua estrangeira: análise de provas de vestibular

Analisa quantitativa e qualitativamente a avaliação da compreensão leitora do Espanhol Língua Estrangeira em provas de vestibular da região norte do Brasil, para assim evidenciar os tipos de operações e, dessa maneira, os tipos de modelos de leitura privilegiados nos testes analisados; estabelecer uma relação entre as provas coletadas e as concepções didáticas subjacentes; identificar problemas nos itens das provas com base em uma série de parâmetros de qualidade; analisar a validade dos testes selecionados; e detectar possíveis diferenças entre as provas elaboradas por Instituições de Ensino Superior públicas e privadas. Essa análise evidencia, por um lado, a tendência a se priorizarem, nessas provas, modelos ascendentes de leitura, com valorização das operações cognitivas de baixo nível e de aspectos léxico-gramaticais da língua, como conseqüência de uma concepção didática de leitura baseada em um enfoque estruturalista. Por outro lado, o estudo empreendido mostra a existência de sérias deficiências nos itens das provas de vestibular, afetando assim sua validade.

Use of cognitive interviewing in the development of a Spanish language genetic knowledge instrument

Study purpose. Genetic advances are significantly impacting healthcare, yet recent studies of ethnic group participation in genetic services demonstrate low utilization rates by Latinos. Limited genetic knowledge is a major barrier. The purpose of this study was to field test items in a Spanish-language instrument that will be used to measure genetic knowledge relevant to type 2 diabetes among members of the ethnically heterogeneous U.S. Latino community. Accurate genetic knowledge measurement can provide the foundation for interventions to enhance genetic service utilization. ^ Design. Three waves of cognitive interviews were conducted in Spanish to field test 44 instrument items Thirty-six Latinos, with 12 persons representative of Mexican, Central and South American, and Cuban heritage participated, including 7 males and 29 females between 22 and 60 years of age; 17 participants had 12 years or less of education. ^ Methods. Text narratives from transcriptions of audiotaped interviews were qualitatively analyzed using a coding strategy to indicate potential sources of response error. Through an iterative process of instrument refinement, codes that emerged from the data were used to guide item revisions at the conclusion of each phase; revised items were examined in subsequent interview waves. ^ Results. Inter-cultural and cross-cultural themes associated with difficulties in interpretation and grammatical structuring of items were identified; difficulties associated with comprehension reflected variations in educational level. Of the original 44 items, 32 were retained, 89% of which were revised. Six additional items reflective of cultural knowledge were constructed, resulting in a 38-item instrument. ^ Conclusions. Use of cognitive interviewing provided a valuable tool for detecting both potential sources of response error and cultural variations in these sources. Analysis of interview data guided successive instrument revisions leading to improved item interpretability and comprehension. Although testing in a larger sample will be essential to test validity and reliability, the outcome of field testing suggests initial content validity of a Spanish-language instrument to measure genetic knowledge relative to type 2 diabetes. ^ Keywords. Latinos, genetic knowledge, instrument development, cognitive interviewing ^

Validation and reliability of the Modified Elderly Mobility Scale

Objective: To establish concurrent validity, interrater and test-retest reliability of the Modified Elderly Mobility Scale (MEMS). Methods: Ninety elderly patients were scored on the MEMS. To establish concurrent validity, 75 patients MEMS scores were compared to Functional Independence Measure (FIM) scores using Spearman's correlation. Videotaped patient performances were used to establish interrater and test-retest reliability using percentage absolute agreement and intraclass correlation coefficients (ICCs). Results: The total MEMS score demonstrated a significant association with the motor (r = 0.725) and total FIM scores (r = 0.718). Absolute agreement for interrater reliability was greater than 93% for all test items, with 97 and 98% for the two new measures, respectively. Test-retest reliability demonstrated similar high levels of absolute agreement and had ICCs ranging from 0.870 to 1.0. Conclusions: The MEMS is a quick, valid and reliable test of motor function of elderly patients with a spread of functional levels.

An investigation into children’s inductive reasoning strategies:What drives the development of category induction?

In a series of studies, I investigated the developmental changes in children’s inductive reasoning strategy, methodological manipulations affecting the trajectory, and driving mechanisms behind the development of category induction. I systematically controlled the nature of the stimuli used, and employed a triad paradigm in which perceptual cues were directly pitted against category membership, to explore under which circumstances children used perceptual or category induction. My induction tasks were designed for children aged 3-9 years old using biologically plausible novel items. In Study 1, I tested 264 children. Using a wide age range allowed me to systematically investigate the developmental trajectory of induction. I also created two degrees of perceptual distractor – high and low – and explored whether the degree of perceptual similarity between target and test items altered children’s strategy preference. A further 52 children were tested in Study 2, to examine whether children showing a perceptual-bias were in fact basing their choice on maturation categories. A gradual transition was observed from perceptual to category induction. However, this transition could not be due to the inability to inhibit high perceptual distractors as children of all ages were equally distracted. Children were also not basing their strategy choices on maturation categories. In Study 3, I investigated category structure (featural vs. relational category rules) and domain (natural vs. artefact) on inductive preference. I tested 403 children. Each child was assigned to either the featural or relational condition, and completed both a natural kind and an artefact task. A further 98 children were tested in Study 4, on the effect of using stimuli labels during the tasks. I observed the same gradual transition from perceptual to category induction preference in Studies 3 and 4. This pattern was stable across domains, but children developed a category-bias one year later for relational categories, arguably due to the greater demands on executive function (EF) posed by these stimuli. Children who received labels during the task made significantly more category choices than those who did not receive labels, possibly due to priming effects. Having investigated influences affecting the developmental trajectory, I continued by exploring the driving mechanism behind the development of category induction. In Study 5, I tested 60 children on a battery of EF tasks as well as my induction task. None of the EF tasks were able to predict inductive variance, therefore EF development is unlikely to be the driving factor behind the transition. Finally in Study 6, I divided 252 children into either a comparison group or an intervention group. The intervention group took part in an interactive educational session at Twycross Zoo about animal adaptations. Both groups took part in four induction tasks, two before and two a week after the zoo visits. There was a significant increase in the number of category choices made in the intervention condition after the zoo visit, a result not observed in the comparison condition. This highlights the role of knowledge in supporting the transition from perceptual to category induction. I suggest that EF development may support induction development, but the driving mechanism behind the transition is an accumulation of knowledge, and an appreciation for the importance of category membership.