Developing a Process to Create and Validate an Instrument Assessing Student Attainment of Competencies at an Intercultural University in Mexico

This study took place at one of the intercultural universities (IUs) of Mexico that serve primarily indigenous students. The IUs are pioneers in higher education despite their numerous challenges (Bertely, 1998; Dietz, 2008; Pineda & Landorf, 2010; Schmelkes, 2009). To overcome educational inequalities among their students (Ahuja, Berumen, Casillas, Crispín, Delgado et al., 2004; Schmelkes, 2009), the IUs have embraced performance-based assessment (PBA; Casillas & Santini, 2006). PBA allows a shared model of power and control related to learning and evaluation (Anderson, 1998). While conducting a review on PBA strategies of the IUs, the researcher did not find a PBA instrument with valid and reliable estimates. The purpose of this study was to develop a process to create a PBA instrument, an analytic general rubric, with acceptable validity and reliability estimates to assess students’ attainment of competencies in one of the IU’s majors, Intercultural Development Management. The Human Capabilities Approach (HCA) was the theoretical framework and a sequential mixed method (Creswell, 2003; Teddlie & Tashakkori, 2009) was the research design. IU participants created a rubric during two focus groups, and seven Spanish-speaking professors in Mexico and the US piloted using students’ research projects. The evidence that demonstrates the attainment of competencies at the IU is a complex set of actual, potential and/or desired performances or achievements, also conceptualized as “functional capabilities” (FCs; Walker, 2008), that can be used to develop a rubric. Results indicate that the rubric’s validity and reliability estimates reached acceptable estimates of 80% agreement, surpassing minimum requirements (Newman, Newman, & Newman, 2011). Implications for practice involve the use of PBA within a formative assessment framework, and dynamic inclusion of constituencies. Recommendations for further research include introducing this study’s instrument-development process to other IUs, conducting parallel mixed design studies exploring the intersection between HCA and assessment, and conducting a case study exploring assessment in intercultural settings. Education articulated through the HCA empowers students (Unterhalter & Brighouse, 2007; Walker, 2008). This study aimed to contribute to the quality of student learning assessment at the IUs by providing a participatory process to develop a PBA instrument.

Developing a process to create and validate an instrument assessing student attainment of competencies at an intercultural university in Mexico

This study took place at one of the intercultural universities (IUs) of Mexico that serve primarily indigenous students. The IUs are pioneers in higher education despite their numerous challenges (Bertely, 1998; Dietz, 2008; Pineda & Landorf, 2010; Schmelkes, 2009). To overcome educational inequalities among their students (Ahuja, Berumen, Casillas, Crispín, Delgado et al., 2004; Schmelkes, 2009), the IUs have embraced performance-based assessment (PBA; Casillas & Santini, 2006). PBA allows a shared model of power and control related to learning and evaluation (Anderson, 1998). While conducting a review on PBA strategies of the IUs, the researcher did not find a PBA instrument with valid and reliable estimates. The purpose of this study was to develop a process to create a PBA instrument, an analytic general rubric, with acceptable validity and reliability estimates to assess students' attainment of competencies in one of the IU's majors, Intercultural Development Management. The Human Capabilities Approach (HCA) was the theoretical framework and a sequential mixed method (Creswell, 2003; Teddlie & Tashakkori, 2009) was the research design. IU participants created a rubric during two focus groups, and seven Spanish-speaking professors in Mexico and the US piloted using students' research projects. The evidence that demonstrates the attainment of competencies at the IU is a complex set of actual, potential and/or desired performances or achievements, also conceptualized as "functional capabilities" (FCs; Walker, 2008), that can be used to develop a rubric. Results indicate that the rubric's validity and reliability estimates reached acceptable estimates of 80% agreement, surpassing minimum requirements (Newman, Newman, & Newman, 2011). Implications for practice involve the use of PBA within a formative assessment framework, and dynamic inclusion of constituencies. Recommendations for further research include introducing this study's instrument-development process to other IUs, conducting parallel mixed design studies exploring the intersection between HCA and assessment, and conducting a case study exploring assessment in intercultural settings. Education articulated through the HCA empowers students (Unterhalter & Brighouse, 2007; Walker, 2008). This study aimed to contribute to the quality of student learning assessment at the IUs by providing a participatory process to develop a PBA instrument.

The development of rubrics to measure undergraduate students' global awareness and global perspective: A validity study

Higher education institutions across the United States have developed global learning initiatives to support student achievement of global awareness and global perspective, but assessment options for these outcomes are extremely limited. A review of research for a global learning initiative at a large, Hispanic-serving, urban, public, research university in South Florida found a lack of instruments designed to measure global awareness and global perspective in the context of an authentic performance assessment. This quasi-experimental study explored the development of two rubrics for the global learning initiative and the extent to which evidence supported the rubrics' validity and reliability. One holistic rubric was developed to measure students' global awareness and the second to measure their global perspective. The study utilized a pretest/posttest nonequivalent group design. Multiple linear regression was used to ascertain the rubrics' ability to discern and compare average learning gains of undergraduate students enrolled in two global learning courses and students enrolled in two non-global learning courses. Parallel pretest/posttest forms of the performance task required students to respond to two open-ended questions, aligned with the learning outcomes, concerning a complex case narrative. Trained faculty raters read responses and used the rubrics to measure students' global awareness and perspective. Reliability was tested by calculating the rates of agreement among raters. Evidence supported the finding that the global awareness and global perspective rubrics yielded scores that were highly reliable measures of students' development of these learning outcomes. Chi-square tests of frequency found significant rates of inter-rater agreement exceeding the study's .80 minimum requirement. Evidence also supported the finding that the rubrics yielded scores that were valid measures of students' global awareness and global perspective. Regression analyses found little evidence of main effects; however, post hoc analyses revealed a significant interaction between global awareness pretest scores and the treatment, the global learning course. Significant interaction was also found between global perspective pretest scores and the treatment. These crossover interactions supported the finding that the global awareness and global perspective rubrics could be used to detect learning differences between the treatment and control groups as well as differences within the treatment group.