Prejudicial evaluation: bias in self-and-peer-assessments of teamwork contributions to design

This chapter considers a simple but important question: can students fairly assess each other’s individual contribution to team designs? The chapter focuses on a key problem when using online self-and-peerassessment to individualising design grades for team assignments, namely rater bias – the possibility of students being biased when assessing their own and their peers’ contributions. Three rater-bias issuesare considered in depth: (1) self-overmarking; (2) gender bias and gender differences; and (3) out-group bias in the peer assessment of international students in multicultural cohorts. Each issue is explored viathe analysis of eight years of quantitative data from the use of an online self-and-peer assessment tool. Evidence is found of self-overmarking and of out-group bias in nonhomogeneous cohorts. However, no evidence is found of gender bias. The chapter concludes with recommendations for design teachers around the assessment of individual contributions to teamwork using self-and-peer assessment.

An extended approach to adjust inconsistent minority peer- and self-assessment scores of teamwork using assessor’s reliability

Teamwork is generally assessed either solely by academic staff or by both academic staff andstudents themselves confidentially as well as collaboratively. Peer- and self-assessments have beenused primarily to assess teamwork process and teacher assessment to assess teamwork product.Peer- and self-assessments are useful to elicit team members’ contribution towards teamwork and toconvert team mark into individual marks, provided the scores are reliable (the extent to which thescores are consistent). However, not all peer- and self-assessment scores are reliable. Anecdotal andliterature evidence suggest that there are several cases of inconsistencies in these scores. Individualcontribution scores given by teammates to an assessee (including himself/herself) can sometimesvary significantly due to both intentional and unintentional reasons. Simply using total individual ratingscores without considering an assessor’s reliability to estimate individual contribution factors cansometime results unfair grades and becomes hindrance to learning through teamwork.PURPOSEThis study proposes an extended approach to adjust inconsistent and/or distorted minority peer andself-assessment scores of teamwork using standard normal probability concept.APPROACHIn order to adjust inconsistent and/or distorted minority peer-and self-assessment scores of teamwork,an extended approach has been proposed. The approach uses the reliability of assessor’s scores ofan assessee using standard normal probability curve. The evaluation of the extended approach isconducted by comparing with the existing approaches using two case examples of peer- and selfassessmentof teamwork where minority team members’ scores are inconsistent.RESULTSThe evaluation of the extended approach shows that the proposed method is superior to the availableapproaches in order to adjust inconsistent peer- and self-assessment scores for special cases wherescores of minority team members are inconsistent. The extended approach helps both to automaticallydetect such scoring anomalies and to adjust the scores so that the fairer contributions to the teamworkwould be obtained and utilised.CONCLUSIONSThe extended approach is useful in that it helps both to automatically detect scoring anomalies and todevise the methods to adjust them. However, the approach does not address the issue of scoringinconsistencies by majority of team members as it uses average score as a basis for identifyinginconsistencies. Moreover, the approach needs to be implemented in the real teamwork environmentin order to identify the impacts of these scoring adjustments in teamwork process and teamworkproduct.

Query routing in cooperative semi-structured peer-to-peer information retrieval networks

Conventional web search engines are centralised in that a single entity crawls and indexes the documents selected for future retrieval, and the relevance models used to determine which documents are relevant to a given user query. As a result, these search engines suffer from several technical drawbacks such as handling scale, timeliness and reliability, in addition to ethical concerns such as commercial manipulation and information censorship. Alleviating the need to rely entirely on a single entity, Peer-to-Peer (P2P) Information Retrieval (IR) has been proposed as a solution, as it distributes the functional components of a web search engine – from crawling and indexing documents, to query processing – across the network of users (or, peers) who use the search engine. This strategy for constructing an IR system poses several efficiency and effectiveness challenges which have been identified in past work. Accordingly, this thesis makes several contributions towards advancing the state of the art in P2P-IR effectiveness by improving the query processing and relevance scoring aspects of a P2P web search. Federated search systems are a form of distributed information retrieval model that route the user’s information need, formulated as a query, to distributed resources and merge the retrieved result lists into a final list. P2P-IR networks are one form of federated search in routing queries and merging result among participating peers. The query is propagated through disseminated nodes to hit the peers that are most likely to contain relevant documents, then the retrieved result lists are merged at different points along the path from the relevant peers to the query initializer (or namely, customer). However, query routing in P2P-IR networks is considered as one of the major challenges and critical part in P2P-IR networks; as the relevant peers might be lost in low-quality peer selection while executing the query routing, and inevitably lead to less effective retrieval results. This motivates this thesis to study and propose query routing techniques to improve retrieval quality in such networks. Cluster-based semi-structured P2P-IR networks exploit the cluster hypothesis to organise the peers into similar semantic clusters where each such semantic cluster is managed by super-peers. In this thesis, I construct three semi-structured P2P-IR models and examine their retrieval effectiveness. I also leverage the cluster centroids at the super-peer level as content representations gathered from cooperative peers to propose a query routing approach called Inverted PeerCluster Index (IPI) that simulates the conventional inverted index of the centralised corpus to organise the statistics of peers’ terms. The results show a competitive retrieval quality in comparison to baseline approaches. Furthermore, I study the applicability of using the conventional Information Retrieval models as peer selection approaches where each peer can be considered as a big document of documents. The experimental evaluation shows comparative and significant results and explains that document retrieval methods are very effective for peer selection that brings back the analogy between documents and peers. Additionally, Learning to Rank (LtR) algorithms are exploited to build a learned classifier for peer ranking at the super-peer level. The experiments show significant results with state-of-the-art resource selection methods and competitive results to corresponding classification-based approaches. Finally, I propose reputation-based query routing approaches that exploit the idea of providing feedback on a specific item in the social community networks and manage it for future decision-making. The system monitors users’ behaviours when they click or download documents from the final ranked list as implicit feedback and mines the given information to build a reputation-based data structure. The data structure is used to score peers and then rank them for query routing. I conduct a set of experiments to cover various scenarios including noisy feedback information (i.e, providing positive feedback on non-relevant documents) to examine the robustness of reputation-based approaches. The empirical evaluation shows significant results in almost all measurement metrics with approximate improvement more than 56% compared to baseline approaches. Thus, based on the results, if one were to choose one technique, reputation-based approaches are clearly the natural choices which also can be deployed on any P2P network.

Using online peer assessment tools to provide feedback and promote learning

A set of slides used for the RAP SIG event on 19 Jan 2017

Sociometric stability and the behavioral correlates of peer acceptance in early childhood

This paper presents findings from an Australian study examining the behavioral correlates and stability of social status for preschool-aged children. The social status of an initial sample of 187 (94 boys and 93 girls) preschool children (mean age 62.4 months, SD = 4.22) was determined through sociometric assessment. Children classified as rejected, neglected and popular (n = 70) were selected for observation. Children were observed for a total of 25 minutes over a three-month period engaging in free play within their preschool centers. Results indicated that children classified as popular were more likely than rejected or neglected children to engage in cooperative play, ongoing connected conversation and to display positive affect. Popular children were less likely than rejected or neglected children to engage in parallel play, onlooker behavior or alone directed behavior. Six months after initial sociometric classification, sociometric interviews were repeated to test for stability and change. Results indicated that preschool-aged children’s social status classifications showed a moderate to high rate of stability for those children classified as popular, rejected and neglected.

Completing the jigsaw : ESL and EFL undergraduate views on interactive peer-based learning

English as a Second Language (ESL) and English as a Foreign Language (EFL) students often face incongruence with Western teaching methods and learning expectations. The aim of this paper is to explore the potential for interactive peer-based learning to engage ESL and EFL language learners provide authentic communication experiences and accelerate learning through two case studies in different contexts. A study was undertaken to investigate student ‘voice’ (Rudduck, 1999, 2005; Rudduck & Flutter, 2004) during an intervention of communicative language teaching using peer-based learning strategies. This article describes unique similarities and subtle differences between ESL and EFL undergraduate learning in two different cultural contexts, using a 'stages of learning matrix' teaching tool to encourage civic skills and self-efficacy. It also suggests ways for teachers to improve on inconsistencies in group-based learning in order to promote more inclusive and congruent learning experiences for English language learners.

Enhancing peer support

A useful way to enhance peer support among young people is to provide peer counsellor training programmes. Research has shown, however, that there are problems associated with typical peer counselling programmes for young people. Most rely on teaching young people counselling skills that are commonly used by professional counsellors when counselling adults and these are not compatible with typical conversation behaviours used by young people. Another problem is that they suggest some commonly used communication processes evident in the conversations of young people are unhelpful, whereas research has shown that this is not the case (Geldard, 2006). In order to overcome these problems, a peer counsellor training programme has been developed for young people as an outcome of research into training young people as peer counsellors (Geldard, 2006).

A pilot study utilising cross-age peer tutoring as a method of intervention for anxious adolescents

Anxiety disorders are the most common psychopathology experienced by young people, with up to 18% of adolescents developing an anxiety disorder. The consequences of these disorders, if left untreated, include impaired peer relationships, school absenteeism and self-concept problems. In addition, anxiety disorders may play a causal role in the development of depression in young people, precede eating disorders and predispose adolescents to substance abuse disorders. While the school is often chosen as a place to provide early intervention for this debilitating disorder, the fact that excessive anxiety is often not recognised in school and that young people are reluctant to seek help, makes identifying these adolescents difficult. Even when these young people are identified, there are problems in providing sensitive programs which are not stigmatising to them within a school setting. One method which may engage this adolescent population could be cross-age peer tutoring. This paper reports on a small pilot study using the “Worrybusters” program and a cross-age peer tutoring method to engage the anxious adolescents. These anxious secondary school students planned activities for teacher-referred anxious primary school students for a term in the high school setting and then delivered those activities to the younger students weekly in the next term in the primary school. Although the secondary school students decreased their scores on anxiety self-report measures there were no significant differences for primary school students’ self-reports. However, the primary school parent reports indicated a significant decrease in their child’s anxiety.

Peer assisted study sessions

My journey with Peer Assisted Study Sessions, or Supplemental Instruction (SI), began in 1993 when I took over a 1st year, 1st semester unit in QUT's Bachelor of Engineering program. The unit had 500 enrolments with students from all 10 engineering majors at QUT. The 500 students received a 2 hour lecture and a 1 hour tutorial per week, usually run by academic staff or postgraduate students. The unit covered basic mechanics, which comprises a challenging set of topics on how forces interact with various bodies. One normally expects 1st year students to find it difficult to come to grips with the material. However, when I ran that unit in 1993, the failure rate had been usually around 50%.

Thinking Collaboratively about the Peer-Review Process for Journal- Article Publication

In this conversation, Kevin K. Kumashiro shares his reflections on challenges to publishing anti-oppressive research in educational journals. He then invites eight current and former editors of leading educational research journals--William F. Pinar, Elizabeth Graue, Carl A. Grant, Maenette K. P. Benham, Ronald H. Heck, James Joseph Scheurich, Allan Luke, and Carmen Luke--to critique and expand on his analysis. Kumashiro begins the conversation by describing his own experiences submitting manuscripts to educational research journals and receiving comments by anonymous reviewers and journal editors. He suggests three ways to rethink the collaborative potential of the peer-review process: as constructive, as multilensed, and as situated. The eight current and former editors of leading educational research journals then critique and expand Kumashiro's analysis. Kumashiro concludes the conversation with additional reflections on barriers and contradictions involved in advancing anti-oppressive educational research in educational journals. (Contains 3 notes.)

Development and pilot test of a peer-support based Cardiac-Diabetes Self-Management program : a study protocol

Background: People with cardiac disease and type 2 diabetes have higher hospital readmission rates (22%)compared to those without diabetes (6%). Self-management is an effective approach to achieve better health outcomes; however there is a lack of specifically designed programs for patients with these dual conditions. This project aims to extend the development and pilot test of a Cardiac-Diabetes Self-Management Program incorporating user-friendly technologies and the preparation of lay personnel to provide follow-up support. Methods/Design: A randomised controlled trial will be used to explore the feasibility and acceptability of the Cardiac-Diabetes Self-Management Program incorporating DVD case studies and trained peers to provide follow-up support by telephone and text-messaging. A total of 30 cardiac patients with type 2 diabetes will be randomised, either to the usual care group, or to the intervention group. Participants in the intervention group will received the Cardiac-Diabetes Self-Management Program in addition to their usual care. The intervention consists of three faceto- face sessions as well as telephone and text-messaging follow up. The face-to-face sessions will be provided by a trained Research Nurse, commencing in the Coronary Care Unit, and continuing after discharge by trained peers. Peers will follow up patients for up to one month after discharge using text messages and telephone support. Data collection will be conducted at baseline (Time 1) and at one month (Time 2). The primary outcomes include self-efficacy, self-care behaviour and knowledge, measured by well established reliable tools. Discussion: This paper presents the study protocol of a randomised controlled trial to pilot evaluates a Cardiac- Diabetes Self-Management program, and the feasibility of incorporating peers in the follow-ups. Results of this study will provide directions for using such mode in delivering a self-management program for patients with both cardiac condition and diabetes. Furthermore, it will provide valuable information of refinement of the intervention program.

Interactive peer-based learning in a comparative case study : what do students gain?

As English increasingly becomes one of the most commonly spoken languages in the world today for a variety of economic, social and cultural reasons, education is impacted by globalisation, the internationalisation of universities and the diversity of learners in classrooms. The challenge for educators is to find more effective ways of teaching English language so that students are better able to create meaning and communicate in the target language as well as to transform knowledge and understanding into relevant skills for a rapidly changing world. This research focuses broadly on English language education underpinned by social constructivist principles informing communicative language teaching and in particular, interactive peer learning approaches. An intervention of interactive peer-based learning in two case study contexts of English as Foreign Language (EFL) undergraduates in a Turkish university and English as Second Language (ESL) undergraduates in an Australian university investigates what students gain from the intervention. Methodology utilising qualitative data gathered from student reflective logs, focus group interviews and researcher field notes emphasises student voice. The cross case comparative study indicates that interactive peer-based learning enhances a range of learning outcomes for both cohorts including engagement, communicative competence, diagnostic feedback as well as assisting development of inclusive social relationships, civic skills, confidence and self efficacy. The learning outcomes facilitate better adaptation to a new learning environment and culture. An iterative instructional matrix tool is a useful product of the research for first year university experiences, teacher training, raising awareness of diversity, building learning communities, and differentiating the curriculum. The study demonstrates that English language learners can experience positive impact through peer-based learning and thus holds an influential key for Australian universities and higher education.

Peer state assessment of TxDOT maintenance program and practices : workshop and road rally findings

In an effort to evaluate and improve their practices to ensure the future excellence of the Texas highway system, the Texas Department of Transportation (TxDOT) sought a forum in which experts from other state departments of transportation could share their expertise. Thus, the Peer State Review of TxDOT Maintenance Practices project was organized and conducted for TxDOT by the Center for Transportation Research (CTR) at The University of Texas at Austin. The goal of the project was to conduct a workshop at CTR and in the Austin District that would educate the visiting peers on TxDOT’s maintenance practices and invite their feedback. CTR and TxDOT arranged the participation of the following directors of maintenance: Steve Takigawa, CA; Roy Rissky, KS; Eric Pitts, GA; Jim Carney, MO; Jennifer Brandenburg, NC; and David Bierschbach, WA. One of the means used to capture the peer reviewers’ opinions was a carefully designed booklet of 15 questions. The peers provided TxDOT with written responses to these questions, and the oral comments made during the workshop were also captured. This information was then compiled and summarized in the following report. An examination of the peers’ comments suggests that TxDOT should use a more holistic, statewide approach to funding and planning rather than funding and planning for each district separately. Additionally, the peers stressed the importance of allocating funds based on the actual conditions of the roadways instead of on inventory. The visiting directors of maintenance also recommended continuing and proliferating programs that enhance communication, such as peer review workshops.

The experience of being a peer outreach volunteer: Benefits and challenges

Peer outreach is an emerging form of non-professional consumer-delivered service in the context of psychiatric rehabilitation. This study identified the benefits and challenges of outreach provision as identified by a group of volunteer outreach workers. One on one semi-structured interviews were carried out with twelve members trained as peer outreach volunteers. Interview transcripts were analysed using a consensual qualitative research approach. Outreach workers typically experienced peer outreach as a positive experience both for themselves and for the recipients. Most found the training and support provided to be appropriate and sufficient. Nonetheless, peer outreach workers did encounter difficulties and sometimes felt need for more training and support. The findings have implications for the development of future peer outreach programs. There is scope for enhanced training and/or supervision and a need for further research to investigate ways to optimise peer outreach.

Peer passenger identity and passenger pressure on young drivers' speeding intentions

Young drivers, aged 17 to 24 years, have the highest fatality rate in Australia. It is believed that part of this risk is due to pressure from peer passengers to engage in speeding; which may be active (i.e., verbal encouragement) or passive (i.e., perceived pressure on the part of the driver). The Theory of Planned Behaviour (TPB) was used to investigate this impact of peer passengers on young drivers, particularly the influence of the type of peer pressure and a driver’s level of identification with their passengers. A scenario-based questionnaire was constructed, informed by focus groups and pilot studies, and distributed to university students (N = 398). The questionnaire measured participants’ intentions and the TPB constructs, including two components of perceived behaviour control, within a baseline scenario as well as an experimental scenario in which the variables of type of pressure and identification were manipulated. Consistent with the hypotheses, the study found that attitudes and self-efficacy significantly predicted intentions over and above the variance explained by the sociodemographic variables of age, gender, self-esteem, sensation seeking, as well as past behaviour and exposure. Across the scenarios, attitudes explained between 4.3% and 14.5%, while self-efficacy to refrain from speeding explained between 4.9% and 17.1%, of the unique variance in intentions to speed. However, contrary to expectations, intentions to speed were found to be higher in the “no passenger” than “passenger present” conditions, although this finding is not completely inconsistent with recent literature. A high level of identification with passengers led to higher intentions to speed than low identification as expected, but, inconsistent with expectations, different types of pressure (i.e., active versus passive) did not influence intentions to speed.