A Summative Evaluation of the Stronger Smarter Learning Communities Project : Vol 1 and Vol 2

This is a summative evaluation of the Stronger Smarter Learning Communities (SSLC) project that examines whether and how the SSLC project had an impact on Australian state schools which adopted its models and approaches. Drawing from qualitative and quantitative data sets, it also presents the largest scale and most comprehensive analysis of Indigenous education practices and outcomes to date. It includes empirical findings on: success in changing school ethos and community engagement; challenges in progress at closure of the 'gap' in conventionally measured achievement and performance; schools' and principals' choices in curriculum and instruction; profiles of teachers' and principals' training and views on teacher education; and a strong emphasis on community and school Indigenoous voices and views on Indigenous education.

Summative evaluation of climate application activities with pastoralists in western Queensland

Survey methods were engaged to measure the change in use and knowledge of climate information by pastoralists in western Queensland. The initial mail survey was undertaken in 2000-01 (n=43) and provided a useful benchmark of pastoralists climate knowledge. Two years of climate applications activities were completed and clients were re-surveyed in 2003 (n=49) to measure the change in knowledge and assess the effectiveness of the climate applications activities. Two methods were used to assess changes in client knowledge, viz., self-assessment and test questions. We found that the use of seasonal climate forecasts in decision making increased from 36% in 2001 (n=42) to 51% in 2003 (n=49) (P=0.07). The self-assessment technique was unsatisfactory as a measure of changing knowledge over short periods (1-3 years), but the test question technique was successful and indicated an improvement in climate knowledge among respondents. The increased levels of use of seasonal climate forecasts in management and improved knowledge was partly attributed to the climate applications activities of the project. Further, those who used seasonal forecasting (n=25) didn't understand key components of forecasts (e.g. probability, median) better than those who didn't use seasonal forecasts (n=24) (P>0.05). This identifies the potential for misunderstanding and misinterpretation of forecasts among users and highlights the need for providers of forecasts to understand the difficulties and prepare simply written descriptions of forecasts and disseminate these with the maps showing probabilities. The most preferred means of accessing climate information were internet, email, 'The Season Ahead' newsletter and newspaper. The least preferred were direct contact with extension officers and attending field days and group meetings. Eighty-six percent of respondents used the internet and 67% used ADSL broadband internet (April 2003). Despite these findings, extension officers play a key role in preparing and publishing the information on the web, in emails and newsletters. We also believe that direct contact with extension officers trained in climate applications is desirable in workshop-like events to improve knowledge of the difficult concepts underpinning climate forecasts, which may then stimulate further adoption.

A summative evaluation of a HIV/AIDS Early Childhood Care, Education and Development Teacher Training Workshop in Mongu, Zambia

Objective. To conduct a summative evaluation of an Early Childhood Care, Education and Development (ECCED) Teacher Training Workshop in Mongu, Zambia by assessing changes in knowledge, attitudes and intent to use the information. ^ Study design. A matched cohort survey design was used with additional qualitative data collected by structured observation of workshop sessions, daily facilitator and participant debriefs and participant interviews. ^ Results. Matching pre and post tests were completed by 27 individuals in addition to daily debriefs, structured workshop observation and participant interviews with 22% of the group. The participant population was predominantly female individuals aged 15-44 years old that had completed high school and additional post-secondary training, been teaching children aged 0 – 8 years for 2-5 years in the Western Province and received other HIV/AIDS and ECCED education. Pre-tests indicated a strong understanding of ECCED principles and misconceptions regarding HIV transmission, prevention and the disease's impact on early childhood development. The workshop was found to significantly increase the participants' knowledge of topics covered by the curriculum (paired t-test, N=27, p = 0.004, 95% CI 1.8, 8.6). Participants began with a more limited understanding of HIV/AIDS than ECCED, but the mean gain was much greater at 7.4 +/- 12.3 points. Significantly more participants believed at post-test that HIV/AIDS education should increase for future educators. The 77.8% of participants that increased their knowledge scores at post-test expressed significantly less fear of having a child with HIV/AIDS in the classroom (Independent Samples t-test, N= 27, p = 0.011). Overall participant fear decreased 15.5%. 92.6% and 88.9% of participants planned at post-test to respectively use and share the taught information in their daily professional lives and reported on innovative strategies to communicate with the community. ^ Conclusions. Teacher training workshops can significantly increase HIV/AIDS awareness and promote positive attitudes in educators working with children affected by HIV/AIDS. Using participant suggested teaching techniques such as poems and songs and translating the materials to the local language could assist future facilitators to both culturally and professionally relate to the workshop audience as well as increase participant capacity to share the information with the local community. ^

NAVET's Boxes - an Evaluation of the Post-Visit Loan Service at a Science Centre in Borås

Many pedagogues believe science centres to be a good complement to the more formal school teaching. For a visit to a science centre to be as educational as possible, there is a need for pre-visit information of some sort, a guided visit, and post-visit work. Many science centres offer loan services of different kinds. At Navet, a science centre in Borås, teachers can borrow boxes with experiments connected to the different themes they provide. The experiments are supposed to be a continuation of the visit and help settle the knowledge gained during the visit. This thesis is an evaluation of how the boxes function in the schools, and what the teachers think of them. The study was conducted through questionnaires and interviews with both teachers and the staff at Navet. The results of the study are very positive. Many teachers have been involved with Navet from the very beginning and they see a visit to Navet as an integrated part of their teaching. Some boxes work better than others and some might need clearer information, but overall the teachers see the boxes as timesavers, as a way to vary their teaching more easily, and as a help for teachers not specialized in mathematics and science.

Evaluation and Model of a Clinical Librarian Program

Clinical medical librarianship is entering its second decade, but little evaluative data has accrued in the literature. Variations from the original programs and novel new approaches have insured the survival of the program so far. The clinical librarian (CL) forms a vital link between the library and the health care professional, operating as an important information transfer agent. However, to further insure the survival of these vital programs, hard evaluative evidence is needed. The University of Texas Medical Branch (UTMB) at Galveston began a CL Program in 1978/79. An extensive three-year pre/post evaluation study was conducted using a specifically developed evaluation model, which, if adopted by others, will provide the needed comparative data. Both a pilot study, or formative evaluation, and a summative evaluation were conducted. The results of this evaluation confirmed many of the conclusions reported by other CL programs. Eight hypotheses were proposed at the beginning of this study. Data were collected and used to support acceptance or rejection of the null hypotheses, and conclusions were drawn according to the results. Implications relevant to the study conclusions and future trends in medical librarianship are also discussed in the closing chapter.

Assessing an experimental approach to industrial policy evaluation:Applying RCT+ to the case of Creative Credits

Experimental methods of policy evaluation are well-established in social policy and development eco-nomics but are rare in industrial and innovation policy. In this paper, we consider the arguments forapplying experimental methods to industrial policy measures, and propose an experimental policy eval-uation approach (which we call RCT+). This approach combines the randomised assignment of firmsto treatment and control groups with a longitudinal data collection strategy incorporating quantitativeand qualitative data (so-called mixed methods). The RCT+ approach is designed to provide a causativerather than purely summative evaluation, i.e. to assess both ‘whether’ and ‘how’ programme outcomesare achieved. In this paper, we assess the RCT+ approach through an evaluation of Creative Credits – aUK business-to-business innovation voucher initiative intended to promote new innovation partnershipsbetween SMEs and creative service providers. The results suggest the potential value of the RCT+ approachto industrial policy evaluation, and the benefits of mixed methods and longitudinal data collection.

Good Practice Report : Blended Learning

In early 2011, the Australian Learning and Teaching Council Ltd (ALTC) commissioned a series of Good Practice Reports on completed ALTC projects and fellowships. This report will: • Provide a summative evaluation of the good practices and key outcomes for teaching and learning from completed ALTC projects and fellowships relating to blended learning • Include a literature review of the good practices and key outcomes for teaching and learning from national and international research • Identify areas in which further work or development are appropriate. The literature abounds with definitions; it can be argued that the various definitions incorporate different perspectives, but there is no single, collectively accepted definition. Blended learning courses in higher education can be placed somewhere on a continuum, between fully online and fully face-to-face courses. Consideration must therefore be given to the different definitions for blended learning presented in the literature and by users and stakeholders. The application of this term in these various projects and fellowships is dependent on the particular focus of the team and the conditions and situations under investigation. One of the key challenges for projects wishing to develop good practice in blended learning is the lack of a universally accepted definition. The findings from these projects and fellowships reveal the potential of blended learning programs to improve both student outcomes and levels of satisfaction. It is clear that this environment can help teaching and learning engage students more effectively and allow greater participation than traditional models. Just as there are many definitions, there are many models and frameworks that can be successfully applied to the design and implementation of such courses. Each academic discipline has different learning objectives and in consequence there can’t be only one correct approach. This is illustrated by the diversity of definitions and applications in the ALTC funded projects and fellowships. A review of the literature found no universally accepted guidelines for good practice in higher education. To inform this evaluation and literature review, the Seven Principles for Good Practice in Undergraduate Education, as outlined by Chickering and Gamson (1987), were adopted: 1. encourages contacts between students and faculty 2. develops reciprocity and cooperation among students 3. uses active learning techniques 4. gives prompt feedback 5. emphasises time on task 6. communicates high expectations 7. respects diverse talents and ways of learning. These blended learning projects have produced a wide range of resources that can be used in many and varied settings. These resources include: books, DVDs, online repositories, pedagogical frameworks, teaching modules. In addition there is valuable information contained in the published research data and literature reviews that inform good practice and can assist in the development of courses that can enrich and improve teaching and learning.

Good Practice Report : Curriculum Renewal

This good practice report, commissioned by the ALTC, provides a summative evaluation of useful outcomes and good practices from ALTC projects and fellowships on curriculum renewal. The report contains: -a summative evaluation of the good practices and key outcomes for teaching and learning from completed ALTC projects and fellowships -a literature review of the good practices and key outcomes for teaching and learning from national and international research -the proposed outcomes and resources for teaching and learning which will be produced by current incomplete ALTC projects and fellowships -identifies areas in which further work or development are appropriate.

Effect and process evaluations of an early childhood obesity prevention intervention

The thesis develops a summative evaluation framework of the NOURISH Randomised Controlled Trial, and applies this to a selection of maternal feeding choice outcomes of the programme. The research is based on an ecological model of the complex set of factors that need to be acted upon to address childhood obesity. The novel approach extends the standard RCT effect evaluation, while also conducting an in-depth evaluation of the process of programme development and implementation. Research findings identify adaptations of intervention delivery that could improve its effectiveness and translatability, in a new cycle of the programme.

Teaching engineering tribology: Elements of assessment design for different learning styles

This study involves teaching engineering students concepts in lubrication engineering that are heavily dependent on mathematics. Excellent learning outcomes have been observed when assessment tasks are devised for a diversity of learning styles. Providing different pathways to knowledge reduces the probability that a single barrier halts progress towards the ultimate learning objective. The interdisciplinary nature of tribology can be used advantageously to tie together multiple elements of engineering to solve real physical problems—an approach that seems to benefit a majority of engineering students. To put this into practice, various assessment items were devised on the study of hydrodynamics, culminating in a project to provide a summative evaluation of the students’ learning achievement. A survey was also conducted to assess other aspects of students’ learning experiences under the headings: ‘perception of learning’ and ‘overall satisfaction’. High degrees of achievement and satisfaction were observed. An attempt has been made to identify the elements contributing to success so that they may be applied to other challenging concepts in engineering education.

Métodos de Avaliação de Aprendizagens em História e Geografia do 3º Ciclo – Um Estudo Comparativo

A avaliação, no contexto de ensino-aprendizagem, ocupa um papel central e fulcral. Subdivide-se em diversas formas e modalidades que nos ajudam a compreender as formas de implementar momentos avaliativos e de conduzir práticas letivas. O presente relatório pretende efetuar uma contextualização teórica de alguns conceitos relacionados com a avaliação – nomeadamente a avaliação sumativa e a avaliação formativa – compreendendo a sua praticabilidade na atividade docente. A Prática de Ensino Supervisionada decorreu em duas escolas diferentes, e a minha prática letiva cingiu-se aos 8º e 9º anos do Ensino Básico. Desta forma, o trabalho apresentado efetua uma comparação entre duas práticas letivas e avaliativas, analisando a sua repercussão ao nível da aprendizagem dos alunos

Assessing evaluative measures for Internet-based courseware

Although there is a consensus in th~ literature on the many uses of the Internet in education, as well as the unique features of the Internet for presenting facts and information, there is no consensus on a standardized method for evaluating Internetbased courseware. Educators rarely have the opportunity to participate in the development of Internet-based courseware, yet they are encouraged to use the technology in their learning environments. This creates a need for summative evaluation methods for Internet-based health courseware. The purpose ofthis study was to assess evaluative measures for Internet-based courseware. Specifically, two entities were evaluated within the study: a) the outcome of the Internet-based courseware, and b) the Internet-based courseware itself. To this end, the Web site www.bodymatters.com was evaluated using two different approaches by two different cohorts. The first approach was a performance appraisal by a group of endusers. A positive, statistically significant change in the students performance was observed due to the intervention ofthe Web site. The second approach was a productoriented evaluation ofthe Web site with the use of a criterion-based checklist and an open-ended comments section. The findings indicate that a summative, criterion-based evaluation is best completed by a multidisciplinary team. The findi~gs also indicated that the two different cohorts reported different product-oriented appraisals of the Web site. The current research confirmed previous research that found that experts returning a poor evaluation of a Web site did not have a relationship to whether or not the end-users performance improved due to the intervention of the Web site.

Akzeptanz und Motivation beim selbstregulierten Lernen unter Einsatz von neuen Medien in der berufsbegleitenden Weiterbildung

Das Ziel der Dissertation war die Untersuchung des Computereinsatzes zur Lern- und Betreuungsunterstützung beim selbstgesteuerten Lernen in der Weiterbildung. In einem bisher konventionell durchgeführten Selbstlernkurs eines berufsbegleitenden Studiengangs, der an das Datenmanagement der Bürodatenverarbeitung heranführt, wurden die Kursunterlagen digitalisiert, die Betreuung auf eine online-basierte Lernbegleitung umgestellt und ein auf die neuen Lernmedien abgestimmtes Lernkonzept entwickelt. Dieses neue Lernkonzept wurde hinsichtlich der Motivation und der Akzeptanz von digitalen Lernmedien evaluiert. Die Evaluation bestand aus zwei Teilen: 1. eine formative, den Entwicklungsprozess begleitende Evaluation zur Optimierung der entwickelten Lernsoftware und des eingeführten Lernkonzeptes, 2. eine sowohl qualitative wie quantitative summative Evaluation der Entwicklungen. Ein zentraler Aspekt der Untersuchung war die freie Wahl der Lernmedien (multimediale Lernsoftware oder konventionelles Begleitbuch) und der Kommunikationsmedien (online-basierte Lernplattform oder die bisher genutzten Kommunikationskanäle: E-Mail, Telefon und Präsenztreffen). Diese Zweigleisigkeit erlaubte eine differenzierte Gegenüberstellung von konventionellen und innovativen Lernarrangements. Die Verbindung von qualitativen und quantitativen Vorgehensweisen, auf Grund derer die subjektiven Einstellungen der Probanden in das Zentrum der Betrachtung rückten, ließen einen Blickwinkel auf den Nutzen und die Wirkung der Neuen Medien in Lernprozessen zu, der es erlaubte einige in der Literatur als gängig angesehene Interpretationen in Frage zu stellen und neu zu diskutieren. So konnten durch eine Kategorisierung des Teilnehmerverhaltens nach online-typisch und nicht online-typisch die Ursache-Wirkungs-Beziehungen der in vielen Untersuchungen angeführten Störungen in Online-Seminaren verdeutlicht werden. In den untersuchten Kursen zeigte sich beispielsweise keine Abhängigkeit der Drop-out-Quote von den Lern- und Betreuungsformen und dass diese Quote mit dem neuen Lernkonzept nur geringfügig beeinflusst werden konnte. Die freie Wahl der Lernmedien führte zu einer gezielten Nutzung der multimedialen Lernsoftware, wodurch die Akzeptanz dieses Lernmedium stieg. Dagegen war die Akzeptanz der Lernenden gegenüber der Lernbegleitung mittels einer Online-Lernplattform von hoch bis sehr niedrig breit gestreut. Unabhängig davon reichte in allen Kursdurchgängen die Online-Betreuung nicht aus, so dass Präsenztreffen erbeten wurde. Hinsichtlich der Motivation war die Wirkung der digitalen Medien niedriger als erwartet. Insgesamt bieten die Ergebnisse Empfehlungen für die Planung und Durchführung von computerunterstützten, online-begleiteten Kursen.

TELMA: Technology enhanced learning environment for minimally invasive surgery

Background: Cognitive skills training for minimally invasive surgery has traditionally relied upon diverse tools, such as seminars or lectures. Web technologies for e-learning have been adopted to provide ubiquitous training and serve as structured repositories for the vast amount of laparoscopic video sources available. However, these technologies fail to offer such features as formative and summative evaluation, guided learning, or collaborative interaction between users. Methodology: The "TELMA" environment is presented as a new technology-enhanced learning platform that increases the user's experience using a four-pillared architecture: (1) an authoring tool for the creation of didactic contents; (2) a learning content and knowledge management system that incorporates a modular and scalable system to capture, catalogue, search, and retrieve multimedia content; (3) an evaluation module that provides learning feedback to users; and (4) a professional network for collaborative learning between users. Face validation of the environment and the authoring tool are presented. Results: Face validation of TELMA reveals the positive perception of surgeons regarding the implementation of TELMA and their willingness to use it as a cognitive skills training tool. Preliminary validation data also reflect the importance of providing an easy-to-use, functional authoring tool to create didactic content. Conclusion: The TELMA environment is currently installed and used at the Jesús Usón Minimally Invasive Surgery Centre and several other Spanish hospitals. Face validation results ascertain the acceptance and usefulness of this new minimally invasive surgery training environment.

CSR and the "underserving":a role for the state, civil society and business?

Purpose – The purpose of this paper is to demonstrate how the conceptual lens of corporate social responsibility (CSR), business and civil society can be used to explore “less popular causes” (in this case, a community-based public sector empirical study of initiatives with offenders) and, in particular, respond to the question used by Walzer “In which society can lives be best led?” Design/methodology/approach – This is a formative and summative evaluation study of a National Offender Management “community payback” offender scheme based in the UK using a mixed method, predominantly qualitative approach that integrates theory and practice. Findings – The paper finds that citizenship actions of front-line public sector employees, working in partnership with other agencies in the community, embody the essence of Walzer's notion of CSR and civil society by going beyond the call of duty to provide additional training and moral support for the community offenders. Originality/value – The paper contributes towards an understanding of how CSR and civil society debates can inform wider aspects of public policy and business through its application to areas of society that are perceived to be “challenging” and “undeserving”.