Formative evaluation to benchmark and improve climate-based decision support for graziers in Western Queelsland

Researchers developing climate-based forecasts, workshops, software tools and information to aid grazier decisions undertook an evaluation study to enhance planning and benchmark impact. One hundred graziers in Western Queensland were randomly selected from 7 shires and surveyed by mail and telephone (43 respondents) to explore levels of knowledge and use of climate information, practices and information needs. We found 36% of respondents apply the Southern Oscillation Index to property decisions but 92% were unaware El Niño Southern Oscillation’s predictive signal in the region is greater for pasture growth than rainfall, suggesting they may not recognise the potential of pasture growth forecasts. Almost 75% of graziers consider they are conservative or risk averse in their attitude to managing their enterprise. Mail respondents (n= 20) if given a 68%, on average, probability of exceeding median rainfall forecast may change a decision; almost two-thirds vary stocking rate based on forage available, last year’s pasture growth or the Southern Oscillation Index; the balance maintain a constant stocking rate strategy; 90% have access to a computer; 75% to the internet and 95% have a fax. This paper presents findings of the study and draws comparisons with a similar study of 174 irrigators in the Northern Murray-Darling Basin (Aust. J. Exp. Ag. 44, 247-257). New insights and information gained are helping the team better understand client needs and plan, design and extend tools and information tailored to grazier knowledge, practice, information needs and preferences. Results have also provided a benchmark against which to measure project impact and have influenced the team to make important changes to their project planning, activities and methods for transferring technology tailored to grazier preferences.

Design and formative evaluation of an information kiosk on cattle health for landless cattle owners

This paper describes and analyses the experience of designing, installing and evaluating a farmer-usable touch screen information kiosk on cattle health in a veterinary institution in Pondicherry. The contents of the kiosk were prepared based on identified demands for information on cattle health, arrived at through various stakeholders meetings. Information on these cattle diseases and conditions affecting the livelihoods of the poor was provided through graphics, text and audio back-up, keeping in mind the needs of landless and illiterate poor cattle owners. A methodology for kiosk evaluation based on the feedback obtained from kiosk facilitator, critical group reflection and individual users was formulated. The formative evaluation reveals the potential strength this ICT has in transferring information to the cattle owners in a service delivery centre. Such information is vital in preventing diseases and helps cattle owners to present and treat their animals at an early stage of disease condition. This in turn helps prevent direct and indirect losses to the cattle owners. The study reveals how an information kiosk installed at a government institution as a freely accessible source of information to all farmers irrespective of their class and caste can help in transfer of information among poor cattle owners, provided periodic updating, interactivity and communication variability are taken care of. Being in the veterinary centre, the kiosk helps stimulate dialogue, and facilitates demand of services based on the information provided by the kiosk screens.

Tapping into Floor Staff: Using the knowledge of floor staff to conduct formative evaluations of exhibits in a Canadian science centre

Most science centres in Canada employ science-educated floor staff to motivate visitorsto have fun while enhancing the educational reach of the exhibits. Although bright andsensitive to visitors’ needs, floor staff are rarely consulted in the planning,implementation, and modification phases of an exhibit. Instead, many developmentteams rely on costly third-party evaluations or skip the front-end and formativeevaluations all together, leading to costly errors that could have been avoided. This studywill seek to reveal a correlation between floor staff’s perception of visitors’ interactionswith an exhibit and visitors’ actual experiences. If a correlation exists, a recommendationcould be made to encourage planning teams to include floor staff in the formative andsummative evaluations of an exhibit. This is especially relevant to science centres withlimited budgets and for whom a divide exists between floor staff and management.In this study, a formative evaluation of one exhibit was conducted, measuring both floorstaff’s perceptions of the visitor experience and visitors’ own perceptions of the exhibit.Floor staff were then trained on visitor evaluation methods. A week later, floor staff andvisitors were surveyed a second time on a different exhibit to determine whether anincrease in accuracy existed.The training session increased the specificity of the motivation and comprehensionresponses and the enthusiasm of the staff, but not their ability to predict observedbehaviours with respect to ergonomics, learning indicators, holding power, and successrates. The results revealed that although floor staff underestimated visitors’ success ratesat the exhibits, staff accurately predicted visitors’ behaviours with respect to holdingpower, ergonomics, learning indicators, motivation and comprehension, both before andafter the staff training.

The mini-clinical evaluation exercise during medical clerkships: are learning needs and learning goals aligned?

OBJECTIVES The generation of learning goals (LGs) that are aligned with learning needs (LNs) is one of the main purposes of formative workplace-based assessment. In this study, we aimed to analyse how often trainer–student pairs identified corresponding LNs in mini-clinical evaluation exercise (mini-CEX) encounters and to what degree these LNs aligned with recorded LGs, taking into account the social environment (e.g. clinic size) in which the mini-CEX was conducted. METHODS Retrospective analyses of adapted mini-CEX forms (trainers’ and students’ assessments) completed by all Year 4 medical students during clerkships were performed. Learning needs were defined by the lowest score(s) assigned to one or more of the mini-CEX domains. Learning goals were categorised qualitatively according to their correspondence with the six mini-CEX domains (e.g. history taking, professionalism). Following descriptive analyses of LNs and LGs, multi-level logistic regression models were used to predict LGs by identified LNs and social context variables. RESULTS A total of 512 trainers and 165 students conducted 1783 mini-CEXs (98% completion rate). Concordantly, trainer–student pairs most often identified LNs in the domains of ‘clinical reasoning’ (23% of 1167 complete forms), ‘organisation/efficiency’ (20%) and ‘physical examination’ (20%). At least one ‘defined’ LG was noted on 313 student forms (18% of 1710). Of the 446 LGs noted in total, the most frequently noted were ‘physical examination’ (49%) and ‘history taking’ (21%). Corresponding LNs as well as social context factors (e.g. clinic size) were found to be predictors of these LGs. CONCLUSIONS Although trainer–student pairs often agreed in the LNs they identified, many assessments did not result in aligned LGs. The sparseness of LGs, their dependency on social context and their partial non-alignment with students’ LNs raise questions about how the full potential of the mini-CEX as not only a ‘diagnostic’ but also an ‘educational’ tool can be exploited.

Alignment between learning needs and learning goals of Mini-CEX in clerkships

Background: Defining learning goals (LG) in alignment with learning needs (LN) is one of the key purposes of formative workplace-based assessment, but studies about this topic are scarce. Summary of Work: We analysed quantitatively and qualitatively how often trainer-student pairs identified the same LN during Mini Clinical Evaluation Exercises (Mini-CEX) in clerkships and to what degree those LNs were in line with the recorded LGs. Multilevel logistic regression models were used to predict LGs by identified LNs, controlling for context variables. Summary of Results: 512 trainers and 165 students conducted 1783 Mini-CEX (98% completion rate). Concordantly, trainer-student pairs most often identified LNs in the domains ‘clinical reasoning’ (23% of 1167 complete forms), ‘organisation / efficiency’ (20%) and ‘physical examination’ (20%). At least one ‘defined’ LG was noted on 313 student forms (18% of 1710), with a total of 446 LGs. Of these, the most frequent LGs were ‘physical examination’ (49% of 446 LGs) and ‘history taking’ (21%); corresponding LNs as well as context variables (e.g. clinic size) were found to be predictors of these LGs. Discussion and Conclusions: Although trainer-student pairs often agreed in their identified LNs, many assessments did not result in an aligned LG or a LG at all. Interventions are needed to enhance the proportion of (aligned) LGs in Mini-CEX in order to tap into its full potential not only as a ‘diagnostic’ but also as an ‘educational tool’. Take-home messages: The sparseness of LGs, their dependency on context variables and their partial non-alignment with students’ LNs raise the question of how the effectiveness of Mini-CEX can be further enhanced.

Formative experience [postgraduate industrial experience]

Producing the graduates that industry wants is more complicated than simply cramming them full of the right knowledge; they must also develop the confidence and understanding to serve the needs of their future employers. This can only be gained from experience in a real manufacturing environment. The authors describe the pioneering approach to industrial experience given to postgraduates at Cranfield University.

Context and models for the analysis of individual and group needs

This paper summarizes the papers presented in the thematic stream Models for the Analysis of Individual and Group Needs, at the 2007 IAEVG-SVP-NCDA Symposium: Vocational Psychology and Career Guidance Practice: An International Partnership. The predominant theme which emerged from the papers was that theory and practice need to be positioned within their contexts. For this paper, context has been formulated as a dimension ranging from the individual’s experience of himself or herself in conversations, including interpersonal transactions and body culture, through to broad higher levels of education, work, nation, and economy.

Formative and summative assessment and the notion of constructive alignment

This paper discusses the different perceptions of first year accounting students about their tutorial activities and their engagements in assessment. As the literature suggests, unless participation in learning activities forms part of graded assessment, it is often difficult to engage students in these activities. Using an action research model, this paper reports the study of first year accounting students' responses to action-oriented learning tasks in tutorials. The paper focuses on the importance of aligning curriculum objectives, learning and teaching activities and assessment, i.e. the notion of constructive alignment. However, as the research findings indicate, without support at institutional level, applying constructive alignment to facilitate quality student learning outcomes is a difficult task. Thus, the impacts of policy constraints on curriculum issues are also discussed, focusing on the limitations faced by tutors and their lack of involvement in curriculum development.