Formative and summative assessment and the notion of constructive alignment

This paper discusses the different perceptions of first year accounting students about their tutorial activities and their engagements in assessment. As the literature suggests, unless participation in learning activities forms part of graded assessment, it is often difficult to engage students in these activities. Using an action research model, this paper reports the study of first year accounting students' responses to action-oriented learning tasks in tutorials. The paper focuses on the importance of aligning curriculum objectives, learning and teaching activities and assessment, i.e. the notion of constructive alignment. However, as the research findings indicate, without support at institutional level, applying constructive alignment to facilitate quality student learning outcomes is a difficult task. Thus, the impacts of policy constraints on curriculum issues are also discussed, focusing on the limitations faced by tutors and their lack of involvement in curriculum development.

Formative and reflective measurement and validation mismatch in survey research : an archival analysis of information systems success constructs 1985-2007

The generic IS-success constructs first identified by DeLone and McLean (1992) continue to be widely employed in research. Yet, recent work by Petter et al (2007) has cast doubt on the validity of many mainstream constructs employed in IS research over the past 3 decades; critiquing the almost universal conceptualization and validation of these constructs as reflective when in many studies the measures appear to have been implicitly operationalized as formative. Cited examples of proper specification of the Delone and McLean constructs are few, particularly in light of their extensive employment in IS research. This paper introduces a four-stage formative construct development framework: Conceive > Operationalize > Respond > Validate (CORV). Employing the CORV framework in an archival analysis of research published in top outlets 1985-2007, the paper explores the extent of possible problems with past IS research due to potential misspecification of the four application-related success dimensions: Individual-Impact, Organizational-Impact, System-Quality and Information-Quality. Results suggest major concerns where there is a mismatch of the Respond and Validate stages. A general dearth of attention to the Operationalize and Respond stages in methodological writings is also observed.

Formative and Summative Assessment and the Notion of Constructive Allignment

This paper discusses perceptions of first year accounting students about their tutorial activities and their engagements in assessment. As the literature suggests, unless participation in learning activities forms part of graded assessment it is often difficult to engage students in these activities. Using an action research model, this paper reports the study of first year accounting students' responses to action-orientated learning tasks in tutorials. The paper focuses on the importance of aligning curriculum objectives, learning and teaching activities and assessment,i.e. the notion of constructive alignment. However, as the research findings indicate, without support at institutional level, applying constructive alignment to facilitate quality student learning outcomes is a difficult task. Thus, the impacts of policy constraints on curriculum issues are also discussed, focusing on the limitations faced by tutors and their lack of involvement in curriculum development.

The Australian Grey Nomads : are they who we think they are? Enhancing formative research through the quantitative assessment of psychological constructs

Issue addressed: Measures of 'social identity' and 'psychological sense of community' were included within a broader formative research inquiry to gain insight into the identity characteristics and level of connectedness among older recreational road travellers (commonly known as Grey Nomads). The research sought to gain insights on how best to reach or speak to this growing driver cohort. ----- ----- Method: Participants included 631 older recreational road travellers ranging in age from 50 years to over 80 years. Data were obtained through three scales which were incorporated into a larger formative research survey; an identity hierarchy, the Three Factor Model of Social Identity and the Sense of Community Index. ----- ----- Results: Older recreational road travellers see themselves principally as couples, with social group identity being secondary. Although many identified to some degree with the Grey Nomad identity, when asked to self categorise as either members of the Broad Network of Recreational Vehicle Travellers or as Grey Nomads, the majority categorised themselves as the former. Those identifying as Grey Nomads, however, reported significantly higher levels of 'social identification' and 'sense of community'. ----- ----- Conclusion: The Grey Nomad identity may not be the best identity at which to target road safety messages for this cohort. Targeting travelling 'couples' may be more efficacious. Using the 'Grey Nomad' identity is likely to reap at least some success, however, given that many identified to some degree with this group identity. Those identifying as Grey Nomads may be more open to community participation or behaviour change given their significantly higher levels of 'social identity' and 'sense of community'.

A formative evaluation of the Stronger Smarter Learning Communities Project. 2011 report.

This report is a formative evaluation of the operations of the DEEWR funded Stronger Smarter Learning Community (SSLC) project from September 2009 to July 2011. It is undertaken by an independent team of researchers from Queensland University of Technology, the University of Newcastle and Harvard University. It reports on findings from: documentary analysis; qualitative case studies of SSLC Hub schools; descriptive, multivariate and multilevel analysis of survey data from school leaders and teachers from SSLC Hub and Affiliate schools and from a control group of non-SSLC schools; and multilevel analysis of school-level data on SSLC Hubs, Affiliates and ACARA like-schools. Key findings from this work are that: • SSLC school leaders and teachers are reporting progress in changing school ethos around issues of: recognition of Indigenous identity, Indigenous leadership, innovative approaches to staffing and school models, Indigenous community engagement and high expectations leadership; • Many Stronger Smarter messages are reportedly having better uptake in schools with high percentages of Indigenous students; • There are no major or consistent patterns of differences between SSLC and non-SSLC schools in teacher and school leader self-reports of curriculum and pedagogy practices; and • There is no evidence to date that SSLC Hubs and Affiliates have increased attendance or increased achievement gains compared to ACARA like-schools. Twenty-one months is relatively early in this school reform project. Hence the major focus of subsequent reports will be on the documentation of comparative longitudinal gains in achievement tests and improved attendance. The 2011 and 2012 research also will model the relationships between change in school ethos/climate, changed Indigenous community relations, improved curriculum/pedagogy, and gains in Indigenous student achievement, attendance and outcomes. The key challenge for SSLC and the Stronger Smarter approach will be whether it can systematically generate change and reform in curriculum and pedagogy practices that can be empirically linked to improved student outcomes.

IS service quality as a multi-dimensional formative constuct

This study is motivated by, and proceeds from, a central interest in the importance of evaluating IS service quality and adopts the IS ZOT SERVQUAL instrument (Kettinger & Lee, 2005) as its core theory base. This study conceptualises IS service quality as a multidimensional formative construct and seeks to answer the main research questions: “Is the IS service quality construct valid as a 1st-order formative, 2nd-order formative multidimensional construct?” Additionally, with the aim of validating the IS service quality construct within its nomological net, as in prior service marketing work, Satisfaction was hypothesised as its immediate consequence. With the goal of testing the above research question, IS service quality and Satisfaction were operationalised in a quantitative survey instrument. Partial least squares (PLS), employing 219 valid responses, largely evidenced the validity of IS service quality as a multidimensional formative construct. The nomological validity of the IS service quality construct was also evidenced by demonstrating that 55% of Satisfaction was explained by the multidimensional formative IS service quality construct.

A formative measurement model of business process model quality

Business process management (BPM) is becoming the dominant management paradigm. Business process modelling is central to BPM, and the resultant business process model the core artefact guiding subsequent process change. Thus, model quality is at the centre, mediating between the modelling effort and related growing investment in ultimate process improvements. Nonetheless, though research interest in the properties that differentiate high quality process models is longstanding, there have been no past reports of a valid, operationalised, holistic measure of business process model quality. In attention to this gap, this paper reports validation of a Business Process Model Quality measurement model, conceptualised as a single-order, formative index. Such a measurement model has value as the dependent variable in rigorously researching the drivers of model quality; as antecedent of ultimate process improvements; and potentially as an economical comparator and diagnostic for practice.

Interpretations of formative assessment in the teaching of English at two Chinese universities : a sociocultural perspective

Formative assessment is increasingly being implemented through policy initiatives in Chinese educational contexts. As an approach to assessment, formative assessment derives many of its key principles from Western contexts, notably through the work of scholars in the UK, the USA and Australia. The question for this paper is the ways that formative assessment has been interpreted in the teaching of College English in Chinese Higher Education. The paper reports on a research study that utilised a sociocultural perspective on learning and assessment to analyse how two Chinese universities – an urban-based Key University and a regional-based Non-Key University – interpreted and enacted a China Ministry of Education policy on formative assessment in College English teaching. Of particular interest for the research were the ways in which the sociocultural conditions of the Chinese context mediated understanding of Western principles and led to their adaptation. The findings from the two universities identified some consistency in localised interpretations of formative assessment which included emphases on process and student participation. The differences related to the specific sociocultural conditions contextualising each university including geographical location, socioeconomic status, and teacher and student roles, expectations and beliefs about English. The findings illustrate the sociocultural tensions in interpreting, adapting and enacting formative assessment in Chinese College English classes and the consequent challenges to and questions about retaining the spirit of formative assessment as it was originally conceptualised.

Using neo-Piagetian theory, formative in-Class tests and think alouds to better understand student thinking : a preliminary report on computer programming

It is acknowledged around the world that many university students struggle with learning to program (McCracken et al., 2001; McGettrick et al., 2005). In this paper, we describe how we have developed a research programme to systematically study and incrementally improve our teaching. We have adopted a research programme with three elements: (1) a theory that provides an organising framework for defining the type of phenomena and data of interest, (2) data on how the class as a whole performs on formative assessment tasks that are framed from within the organising framework, and (3) data from one-on-one think aloud sessions, to establish why students struggle with some of those in-class formative assessment tasks. We teach introductory computer programming, but this three-element structure of our research is applicable to many areas of engineering education research.

A critical evaluation and comparison of two formative measures of system quality using criterion variables

Evaluating the validity of formative variables has presented ongoing challenges for researchers. In this paper we use global criterion measures to compare and critically evaluate two alternative formative measures of System Quality. One model is based on the ISO-9126 software quality standard, and the other is based on a leading information systems research model. We find that despite both models having a strong provenance, many of the items appear to be non-significant in our study. We examine the implications of this by evaluating the quality of the criterion variables we used, and the performance of PLS when evaluating formative models with a large number of items. We find that our respondents had difficulty distinguishing between global criterion variables measuring different aspects of overall System Quality. Also, because formative indicators “compete with one another” in PLS, it may be difficult to develop a set of measures which are all significant for a complex formative construct with a broad scope and a large number of items. Overall, we suggest that there is cautious evidence that both sets of measures are valid and largely equivalent, although questions still remain about the measures, the use of criterion variables, and the use of PLS for this type of model evaluation.

The enactment of formative assessment in English language classrooms in two Chinese universities : teacher and student responses

The College English Curriculum Requirements (CECR), announced by the Chinese Ministry of Education in 2007, recommended the inclusion of formative assessment into the existing summative assessment framework of College English. This policy had the potential to fundamentally change the nature of assessment and its role in the teaching and learning of English in Chinese universities. In order to document and analyse these changes, case studies involving English language teachers and learners were undertaken in two Chinese Universities: one a Key university in the national capital; the other a non-Key university in a western province. The case study design incorporated classroom observations and interviews with English language teachers and their students. The type and focus of feedback and the engagement of students in assessment were analysed in the two contexts. Fundamental to the analysis was the concept of enactment, with the focus of this study on the ways that policy ideas and principles were enacted in the practices of the Chinese university classroom. Understandings of formative assessment as applied in contexts other than the predominantly Western, Anglophone contexts from where many of its principles derive, are offered.

The OO files : providing online formative feedback on Contract Law learning in a narrative setting

Feedback, both formative and summative, enables students to reflect on their understandings and to restructure their thinking to develop their capabilities. It can also encourage positive motivation and help boost self-esteem. Online multiple choice questions can be an efficient and effective means of providing timely formative feedback. At the same time, locating learning in a narrative environment can facilitate engaging and effective learning experiences. Narratives can help learners to navigate through information and support cognitive and imaginative engagement. This article will discuss The 00 Files, an online suite of modules containing multiple choice questions situated in the narrative of a fictional law firm. It notes student responses to the program and discusses lessons that may be learnt from its development which may be of assistance to academics considering the development of similar programs for their courses.

Using formative feedback to identify and support first year chemistry students with missing or misconceptions. A Practice Report

Students entering tertiary studies possess a diverse range of prior experiences in their academic preparation for tertiary chemistry so academics need tools to enable them to respond to issues in diversity in conceptual models possessed by entering students. Concept inventories can be used to provide formative feedback to help students identify concepts that they need to address to improve construction of subsequent understanding enabling their learning. Modular, formative learning activities that can be administered inside or outside of class in first year chemistry courses have been developed. These activities address key missing and mis-conceptions possessed by incoming student. Engagement in these learning activities by students and academics will help shift the culture of diagnostic and formative assessment within the tertiary context and address issues around the secondary/tertiary transition. This diagnostic/intervention framework is currently being trialed across five Australian tertiary institutions encompassing a large heterogeneous sample of students.

Formative evaluation for improving collaborative planning : a case study at the regional scale

In an earlier paper (Cameron & Johnson 2004) we introduced the idea of formative evaluation (or evaluation for development), the purpose of which is to provide information for improving planning programs and activities. This type of evaluation differs from the two other types: outcome evaluation which aims to judge the success or otherwise of a program; and evaluation for knowledge which seeks to contribute to theoretical work on planning processes and activities. In the earlier paper we also outlined the first stage of formative evaluation in the SEQ 2021 regional planning exercise showing how the process of planning for community engagement was modified in light of the evaluation findings. This current paper details the second stage of formative evaluation in which the collaborative planning component of SEQ 2021 was evaluated, as such it further demonstrates how formative evaluation can be used to improve planning programs. The evaluation findings also provide insights into strategies for more effective collaborative planning. We begin with an overview of collaborative approaches to regional planning, including the SEQ 2021 regional planning program. We then outline formal and informal evaluations of various collaborative regional planning exercises, including the predecessor of SEQ 2021 - SEQ 2001. This sets the scene for discussion of the approach used to evaluate the collaborative component of SEQ 2021. After outlining the main findings from the evaluation and the ways these findings were used to refine the collaborative planning process we conclude with a series of recommendations, relevant not only to SEQ 2021 but to other collaborative planning exercises