Increasing the rigour and trustworthiness of participatory evaluations : learnings from the field

Participatory evaluation and participatory action research (PAR) are increasingly used in community-based programs and initiatives and there is a growing acknowledgement of their value. These methodologies focus more on knowledge generated and constructed through lived experience than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that acknowledges and takes into account the diverse perspectives, values and interpretations of participants and evaluation professionals. However, evaluation rigour need not be lost in this approach. Increasing the rigour and trustworthiness of participatory evaluations and PAR increases the likelihood that results are seen as credible and are used to continually improve programs and policies.----- Drawing on learnings and critical reflections about the use of feminist and participatory forms of evaluation and PAR over a 10-year period, significant sources of rigour identified include:----- • participation and communication methods that develop relations of mutual trust and open communication----- • using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection----- • ongoing meta-evaluation and critical reflection----- • critically assessing the intended and unintended impacts of evaluations, using relevant theoretical models----- • using rigorous data analysis and reporting processes----- • participant reviews of evaluation case studies, impact assessments and reports.

Communication and culture : mobile telephony in PNG villages

This paper examines the recent introduction of mobile telephony into rural communities in Papua New Guinea (PNG). It presents the findings of substantial fieldwork conducted in 2009, and suggests ways in which the new technology is already changing people’s lives and relationships. The paper identifies the roles of mobile telephones in two communities, the changes taking place and how villagers are responding to them. Comparison of the two villages is strategic as it highlights similarities in perceptions of mobile phones in these two very different settings. An ethnographic approach is adopted, situated within an interpretative methodology. Data collection methods include semi-structured interviews, orally-administered surveys and participant observation. The village lifestyle or ‘culture’ provides an important lens for understanding this data and the assertions made by village respondents. This research is significant as it addresses changes currently occurring in the communication methods of whole communities.

Communication patterns within social networks : a case study of Australian women

Based on the perception that “individual bonds to one another is the essence of society” (Fischer, 1982a, p. 2), this paper examines contemporary networks of friends: friendworks, of adult women in an Australian sea change community. Communication patterns are examined drawing on findings from a case study of 26 women aged 35-76 years. Among the case study participants, many have undertaken a ‘sea change’ as adults, which in most cases has led to a significant reconstruction of their friendworks. Location and lifestyle are identified as impacting factors on communication patterns with friends; face-to-face interactions are by far the most frequent and preferred method of communication among the participants. The landline telephone and internet are the main communication methods used to maintain friendships with distant loved ones, while the mobile phone is reported as the communication method employed the least. The infrequency of mobile phone use can be attributed to cost issues, highlighting a discrepancy between these women’s social and communication needs and the current Australian mobile phone policy.

Communicating friendships : a case study of women in an Australian 'seachange' town

This thesis proposes =friendworks‘ as an important sub-group of social networks, comprised of networks of friends. It investigates friendworks of a particular group of adult Australian women as a way of understanding neglected aspects of social networking practices. Friendworks are contextualised to highlight two main themes of interest: population mobility and communication practices. The impact of relocation on individuals, local communities and the wider society is explored through a case study of female friendworks in a seachange community. Research findings point to the importance of friendworks in building and cohering social and emotional support, well-being, belonging and senses of place and community. Different types of communication methods were used by research participants for mediating different kinds of social ties within the friendworks considered here. Communication patterns were influenced by geographical proximity to friends, and the type of social support required of them (emotional, instrumental or companionship). Most findings were consistent with broader social patterns of communication. For example, face-to-face interactions were the dominant and most favoured communication method between local friends, regardless of whether they were weak or strong ties. The fixed-telephone and the internet were commonly in use to maintain old and geographically distant social ties, while mobile phones were used the least among friends in comparison with other communication methods. The key finding of this thesis is that friendworks are an extremely important solid network in contemporary society, providing mooring relations in a mobile world. Paradoxically, however, for women in this study, the mobile phone, which is popularly perceived as a flexible, multi-purpose communication technology for people on the move, was the least versatile of all technologies for maintaining friendworks. The cost of services was the main inhibitor here. The internet was found to be the most versatile communication technology and was used to support various types of social ties: strong, weak, local and distant. This thesis also highlights the value of the concept of friendworks as well as networks for communication research and policy investigating individuals‘ motivations and practices.

The impact of communication technologies on life and relationship satisfaction

Previous studies have shown a relationship between the use of communications technology and well-being, particularly mediated through its effect on personal relationships. However, there is some debate over whether this effect is positive or negative. The present study explored this issue further, examining whether the effect varies depending on the type of communications technology, and the nature of the personal relationship. An online survey was conducted with 3,421 participants in three countries (Australia, UK and US). It examined the use of ten communication methods, overall satisfaction with life and satisfaction with four different kinds of relationships (close and extended family, and close and distant friends). Results indicate that richer communication methods, which include non-verbal cues, were positively associated with both overall satisfaction with life and satisfaction with relationships. These methods included face-to-face communication, and phone and video calls. Conversely, more restricted methods, such as text messaging and instant messaging, were negatively associated with both variables. Social networking was negatively associated with overall satisfaction, but not with satisfaction with relationships. The strength of the association between a communications method and satisfaction with a relationship varied depending on the type of relationship, but whether it was positive or negative did not change.

Monitoring animal behaviour and environmental interactions : GPS collars and satellite remote sensing within a satellite remote sensing context

Remote monitoring of animal behaviour in the environment can assist in managing both the animal and its environmental impact. GPS collars which record animal locations with high temporal frequency allow researchers to monitor both animal behaviour and interactions with the environment. These ground-based sensors can be combined with remotely-sensed satellite images to understand animal-landscape interactions. The key to combining these technologies is communication methods such as wireless sensor networks (WSNs). We explore this concept using a case-study from an extensive cattle enterprise in northern Australia and demonstrate the potential for combining GPS collars and satellite images in a WSN to monitor behavioural preferences and social behaviour of cattle.

Monitoring animal behaviour and environmental interactions using wireless sensor networks, GPS collars and satellite remote sensing

Remote monitoring of animal behaviour in the environment can assist in managing both the animal and its environmental impact. GPS collars which record animal locations with high temporal frequency allow researchers to monitor both animal behaviour and interactions with the environment. These ground-based sensors can be combined with remotely-sensed satellite images to understand animal-landscape interactions. The key to combining these technologies is communication methods such as wireless sensor networks (WSNs). We explore this concept using a case-study from an extensive cattle enterprise in northern Australia and demonstrate the potential for combining GPS collars and satellite images in a WSN to monitor behavioural preferences and social behaviour of cattle.

Out of touch? On the lack of emotional support over the mobile phone

Findings from an Australian case study of adult women expose general, light and basic use of mobile phones. Participants used their mobile phone mainly for coordination and to a lesser extent for practicing intrinsic interactions motivated by emotional support purposes. This paper focuses on social and emotional support over the mobile phone. Though crucial to individuals, emotional support seems to be a neglected area of research among mobile communication studies, all the more so when focusing on adult women. This study addresses this literature gap. The empirical findings are based on a case study of 26 women over 35 years of age residing in one coastal Australian town. The research design included a combination of quantitative and qualitative methods. This paper examines the communication methods adult women use for social and emotional support, and analyses reasons and social implications of this limited intrinsic communication use pattern over the mobile phone.

Infinite-horizon policy-gradient estimation

Gradient-based approaches to direct policy search in reinforcement learning have received much recent attention as a means to solve problems of partial observability and to avoid some of the problems associated with policy degradation in value-function methods. In this paper we introduce GPOMDP, a simulation-based algorithm for generating a biased estimate of the gradient of the average reward in Partially Observable Markov Decision Processes (POMDPs) controlled by parameterized stochastic policies. A similar algorithm was proposed by Kimura, Yamamura, and Kobayashi (1995). The algorithm's chief advantages are that it requires storage of only twice the number of policy parameters, uses one free parameter β ∈ [0,1) (which has a natural interpretation in terms of bias-variance trade-off), and requires no knowledge of the underlying state. We prove convergence of GPOMDP, and show how the correct choice of the parameter β is related to the mixing time of the controlled POMDP. We briefly describe extensions of GPOMDP to controlled Markov chains, continuous state, observation and control spaces, multiple-agents, higher-order derivatives, and a version for training stochastic policies with internal states. In a companion paper (Baxter, Bartlett, & Weaver, 2001) we show how the gradient estimates generated by GPOMDP can be used in both a traditional stochastic gradient algorithm and a conjugate-gradient procedure to find local optima of the average reward. ©2001 AI Access Foundation and Morgan Kaufmann Publishers. All rights reserved.

Early experience of mobile telephony : a comparison of two villages in Papua New Guinea

This paper examines social change following the recent introduction of mobile telephony into rural communities in Papua New Guinea (PNG). It presents the findings of substantial fieldwork conducted in 2009, and suggests ways in which the new technology is already changing people’s lives and relationships. The paper identifies the roles of mobile telephones in two communities, the changes taking place, and how villagers are responding to them. Comparison of the two villages is strategic as it highlights similarities in perceptions of mobile phones in these two very different settings. An ethnographic approach is adopted, situated within an interpretative methodology. Data collection methods include semi-structured interviews, orally-administered surveys and participant observation. The theoretical lens is focused on the ‘communicative ecology’ concept, which stems from the communication research tradition. This research is significant as it addresses changes currently occurring in the communication methods of whole communities.

Editorial of "International Journal of Critical Indigenous Studies Volume 4, Number 2, 2011"

This special edition of the International Journal of Critical Indigenous Studies focuses upon the work scholars within the growing discipline of Aboriginal and Torres Strait Islander health studies. The lamentable state of Indigenous health in Australia is reflected in Indigenous populations elsewhere, especially where settler colonialism has left an indelible mark. This special edition therefore speaks to where Indigenous health outcomes and the efficacy of remedies are causing concern. Common to all is the demand that Indigenous people are placed front and centre of all attempts to improve health outcomes and that improvements are sought in culturally sensitive ways. Terry Dunbar presents findings from a research study that set out to investigate the Indigenous experiences of health and family services in the Northern Territory, Australia. The study asserts that cultural security is an integral and vital element of any policy that will impact upon Indigenous peoples. Dunbar concludes by arguing that in seeking positive change with regard to cultural security or otherwise, the most vociferous champions of that change are likely to be the Aboriginal communities affected. The article by Bronwyn Fredericks, Karen Adams, Sandra Angus and Melissa Walker also highlights the need to involve Aboriginal and Torres Strait Islander people, in this case women, in the design and development of strategies affecting their lives. Utilising routine communication methods such the ‘talking circle’ and the process referred to as ‘talkin’ up’, where women ‘talk back’ to one another about issues of personal importance, the article argues that the health strategy which emerged through these consultation approaches was more accurately informed by and responsive to women’s health need. Indeed, the resulting strategy reflected the women’s sense of themselves and the clear direction they felt their health services and polices should take.

Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal

Developing an effective impact evaluation framework, managing and conducting rigorous impact evaluations, and developing a strong research and evaluation culture within development communication organisations presents many challenges. This is especially so when both the community and organisational context is continually changing and the outcomes of programs are complex and difficult to clearly identify.----- This paper presents a case study from a research project being conducted from 2007-2010 that aims to address these challenges and issues, entitled Assessing Communication for Social Change: A New Agenda in Impact Assessment. Building on previous development communication projects which used ethnographic action research, this project is developing, trailing and rigorously evaluating a participatory impact assessment methodology for assessing the social change impacts of community radio programs in Nepal. This project is a collaboration between Equal Access – Nepal (EAN), Equal Access – International, local stakeholders and listeners, a network of trained community researchers, and a research team from two Australian universities. A key element of the project is the establishment of an organisational culture within EAN that values and supports the impact assessment process being developed, which is based on continuous action learning and improvement. The paper describes the situation related to monitoring and evaluation (M&E) and impact assessment before the project began, in which EAN was often reliant on time-bound studies and ‘success stories’ derived from listener letters and feedback. We then outline the various strategies used in an effort to develop stronger and more effective impact assessment and M&E systems, and the gradual changes that have occurred to date. These changes include a greater understanding of the value of adopting a participatory, holistic, evidence-based approach to impact assessment. We also critically review the many challenges experienced in this process, including:----- • Tension between the pressure from donors to ‘prove’ impacts and the adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations.----- • Resistance from the content teams to changing their existing M&E practices and to the perceived complexity of the approach.----- • Lack of meaningful connection between the M&E and content teams.----- • Human resource problems and lack of capacity in analysing qualitative data and reporting results.----- • The contextual challenges, including extreme poverty, wide cultural and linguistic diversity, poor transport and communications infrastructure, and political instability.----- • A general lack of acceptance of the importance of evaluation within Nepal due to accepting everything as fate or ‘natural’ rather than requiring investigation into a problem.

The use of income valuation methods to value rural property

Despite the advances that have been made in relation to the valuation of commercial, industrial and retail property, there has not been the same progress in relation to the valuation of rural property. Although the majority of rural property valuations also require the valuer to carry out a full analysis of the economic performance of the farming operations, this information is rarely used to assess the value of the property, nor is it even used for a secondary valuation method. Over the past 20 years the nature of rural valuation practice has required rural valuers to undertake studies in both agriculture (farm management) and valuation, especially if carrying out valuation work for financial institutions. The additional farm financial information obtained by rural valuers exceeds that level of information required to value commercial, retail and industrial by the capitalisation of net rent/profit valuation method and is very similar to the level of information required for the valuation of commercial and retail property by the Discounted Cash Flow valuation method. On this basis the valuers specialising in rural valuation practice should have the necessary skills and information to value rural properties by an income valuation method. Although the direct comparison method of valuation has been sufficient in the past to value rural properties the future use of the method as the main valuation method is limited and valuers need to adopt an income valuation method as at least a secondary valuation method to overcome the problems associated with the use of direct comparison as the only rural property valuation method. This paper will review the results of an extensive survey carried out by rural property valuers in New South Wales (NSW), Australia, in relation to the impact of farm management on rural property values and rural property income potential.

IT-enhanced communication protocols for building project management

Purpose – Building project management requires real time flow of information between all the project team members or the supply chain members. In the present scenario, when project participants are geographically separated, adoption of Information Communication Technology (ICT) enables such effective communication. But strategic adoption of ICT requires that all the supply chain members follow the accepted methods of communication or the communication protocols. The majority of the construction organizations are small and medium enterprises (SMEs). This research, therefore, proposes to focus on developing IT-enhanced communication protocols for building project management by SMEs. Design/methodology/approach – The research adopts a sequential mixed methods approach, where data collection and analysis are conducted in both the quantitative and qualitative phases of research. Findings – The protocols are proposed as a “Strategic Model for Enhancing ICT Diffusion in Building Projects”. The framework for the model is discussed at three levels of study, i.e industry, organization, and people. Practical implications – While the research was conducted in an Indian context, the research outcome is envisaged to be widely applicable in other countries with due considerations. Originality/value – The developed framework has implications for national level bodies and academic institutions, organizations, people or project managers and is applicable at the international level after due considerations.

Evaluating Communication for Development : A Framework for Social Change

Evaluating Communication for Development presents a comprehensive framework for evaluating communication for development (C4D). This framework combines the latest thinking from a number of fields in new ways. It critiques dominant instrumental, accountability-based approaches to development and evaluation and offers an alternative holistic, participatory, mixed methods approach based on systems and complexity thinking and other key concepts. It maintains a focus on power, gender and other differences and social norms. The authors have designed the framework as a way to focus on achieving sustainable social change and to continually improve and develop C4D initiatives. The benefits and rigour of this approach are supported by examples and case studies from a number of action research and evaluation capacity development projects undertaken by the authors over the past fifteen years. Building on current arguments within the fields of C4D and development, the authors reinforce the case for effective communication being a central and vital component of participatory forms of development, something that needs to be appreciated by decision makers. They also consider ways of increasing the effectiveness of evaluation capacity development from grassroots to management level in the development context, an issue of growing importance to improving the quality, effectiveness and utilisation of monitoring and evaluation studies in this field. The book includes a critical review of the key approaches, methodologies and methods that are considered effective for planning evaluation, assessing the outcomes of C4D, and engaging in continuous learning. This rigorous book is of immense theoretical and practical value to students, scholars, and professionals researching or working in development, communication and media, applied anthropology, and evaluation and program planning.