Developing a participatory impact assessment approach and action research culture within a communication for social change organisation in Nepal

Developing an effective impact evaluation framework, managing and conducting rigorous impact evaluations, and developing a strong research and evaluation culture within development communication organisations presents many challenges. This is especially so when both the community and organisational context is continually changing and the outcomes of programs are complex and difficult to clearly identify.----- This paper presents a case study from a research project being conducted from 2007-2010 that aims to address these challenges and issues, entitled Assessing Communication for Social Change: A New Agenda in Impact Assessment. Building on previous development communication projects which used ethnographic action research, this project is developing, trailing and rigorously evaluating a participatory impact assessment methodology for assessing the social change impacts of community radio programs in Nepal. This project is a collaboration between Equal Access – Nepal (EAN), Equal Access – International, local stakeholders and listeners, a network of trained community researchers, and a research team from two Australian universities. A key element of the project is the establishment of an organisational culture within EAN that values and supports the impact assessment process being developed, which is based on continuous action learning and improvement. The paper describes the situation related to monitoring and evaluation (M&E) and impact assessment before the project began, in which EAN was often reliant on time-bound studies and ‘success stories’ derived from listener letters and feedback. We then outline the various strategies used in an effort to develop stronger and more effective impact assessment and M&E systems, and the gradual changes that have occurred to date. These changes include a greater understanding of the value of adopting a participatory, holistic, evidence-based approach to impact assessment. We also critically review the many challenges experienced in this process, including:----- • Tension between the pressure from donors to ‘prove’ impacts and the adoption of a bottom-up, participatory approach based on ‘improving’ programs in ways that meet community needs and aspirations.----- • Resistance from the content teams to changing their existing M&E practices and to the perceived complexity of the approach.----- • Lack of meaningful connection between the M&E and content teams.----- • Human resource problems and lack of capacity in analysing qualitative data and reporting results.----- • The contextual challenges, including extreme poverty, wide cultural and linguistic diversity, poor transport and communications infrastructure, and political instability.----- • A general lack of acceptance of the importance of evaluation within Nepal due to accepting everything as fate or ‘natural’ rather than requiring investigation into a problem.

Finding order in chaos : a problem based collaborative assignment for 1300 students across 19 disciplines

The use of collaborative assignments for assessment is a risky undertaking for students and course designers. Yet the benefits, in terms of core learning outcomes, competencies, collaborative sense making and student involvement, suggest that the effort is worthwhile. Formal descriptions and rules do little to ameliorate the perception of risk and increased anxiety by students. (Ryan, 2007). BEB100 Introducing Professional Learning is a faculty-wide foundation unit with over 1300 students from 19 disciplines across the Faculty of the Built Environment and Engineering (“BEE”) at the Queensland University of Technology (“QUT”), Brisbane, Australia. Finding order in chaos outlines the approach and justification, assessment criteria, learning resources, teamwork tools, tutorial management, communication strategies, 2007-09 Student Learning Experience Survey results, annual improvements, findings and outcomes.

Experience of problem-based learning (PBL) in virtual space : a phenomenographical study

This thesis reports the outcomes of an investigation into students’ experience of Problem-based learning (PBL) in virtual space. PBL is increasingly being used in many fields including engineering education. At the same time many engineering education providers are turning to online distance education. Unfortunately there is a dearth of research into what constitutes an effective learning experience for adult learners who undertake PBL instruction through online distance education. Research was therefore focussed on discovering the qualitatively different ways that students experience PBL in virtual space. Data was collected in an electronic environment from a course, which adopted the PBL strategy and was delivered entirely in virtual space. Students in this course were asked to respond to open-ended questions designed to elicit their learning experience in the course. Data was analysed using the phenomenographical approach. This interpretative research method concentrated on mapping the qualitative differences in students’ interpretations of their experience in the course. Five qualitatively different ways of experiencing were discovered: Conception 1: ‘A necessary evil for program progression’; Conception 2: ‘Developing skills to understand, evaluate, and solve technical Engineering and Surveying problems’; Conception 3: ‘Developing skills to work effectively in teams in virtual space’; Conception 4: ‘A unique approach to learning how to learn’; Conception 5: ‘Enhancing personal growth’. Each conception reveals variation in how students attend to learning by PBL in virtual space. Results indicate that the design of students’ online learning experience was responsible for making students aware of deeper ways of experiencing PBL in virtual space. Results also suggest that the quality and quantity of interaction with the team facilitator may have a significant impact on the student experience in virtual PBL courses. The outcomes imply pedagogical strategies can be devised for shifting students’ focus as they engage in the virtual PBL experience to effectively manage the student learning experience and thereby ensure that they gain maximum benefit. The results from this research hold important ramifications for graduates with respect to their ease of transition into professional work as well as their later professional competence in terms of problem solving, ability to transfer basic knowledge to real-life engineering scenarios, ability to adapt to changes and apply knowledge in unusual situations, ability to think critically and creatively, and a commitment to continuous life-long learning and self-improvement.

Abstracting and correlating heterogeneous events to detect complex scenarios

The research presented in this thesis addresses inherent problems in signaturebased intrusion detection systems (IDSs) operating in heterogeneous environments. The research proposes a solution to address the difficulties associated with multistep attack scenario specification and detection for such environments. The research has focused on two distinct problems: the representation of events derived from heterogeneous sources and multi-step attack specification and detection. The first part of the research investigates the application of an event abstraction model to event logs collected from a heterogeneous environment. The event abstraction model comprises a hierarchy of events derived from different log sources such as system audit data, application logs, captured network traffic, and intrusion detection system alerts. Unlike existing event abstraction models where low-level information may be discarded during the abstraction process, the event abstraction model presented in this work preserves all low-level information as well as providing high-level information in the form of abstract events. The event abstraction model presented in this work was designed independently of any particular IDS and thus may be used by any IDS, intrusion forensic tools, or monitoring tools. The second part of the research investigates the use of unification for multi-step attack scenario specification and detection. Multi-step attack scenarios are hard to specify and detect as they often involve the correlation of events from multiple sources which may be affected by time uncertainty. The unification algorithm provides a simple and straightforward scenario matching mechanism by using variable instantiation where variables represent events as defined in the event abstraction model. The third part of the research looks into the solution to address time uncertainty. Clock synchronisation is crucial for detecting multi-step attack scenarios which involve logs from multiple hosts. Issues involving time uncertainty have been largely neglected by intrusion detection research. The system presented in this research introduces two techniques for addressing time uncertainty issues: clock skew compensation and clock drift modelling using linear regression. An off-line IDS prototype for detecting multi-step attacks has been implemented. The prototype comprises two modules: implementation of the abstract event system architecture (AESA) and of the scenario detection module. The scenario detection module implements our signature language developed based on the Python programming language syntax and the unification-based scenario detection engine. The prototype has been evaluated using a publicly available dataset of real attack traffic and event logs and a synthetic dataset. The distinct features of the public dataset are the fact that it contains multi-step attacks which involve multiple hosts with clock skew and clock drift. These features allow us to demonstrate the application and the advantages of the contributions of this research. All instances of multi-step attacks in the dataset have been correctly identified even though there exists a significant clock skew and drift in the dataset. Future work identified by this research would be to develop a refined unification algorithm suitable for processing streams of events to enable an on-line detection. In terms of time uncertainty, identified future work would be to develop mechanisms which allows automatic clock skew and clock drift identification and correction. The immediate application of the research presented in this thesis is the framework of an off-line IDS which processes events from heterogeneous sources using abstraction and which can detect multi-step attack scenarios which may involve time uncertainty.

Managing infrastructure transitions : a complex adaptive systems perspective

Engineering assets such as roads, rail, bridges and other forms of public works are vital to the effective functioning of societies {Herder, 2006 #128}. Proficient provision of this physical infrastructure is therefore one of the key activities of government {Lædre, 2006 #123}. In order to ensure engineering assets are procured and maintained on behalf of citizens, government needs to devise the appropriate policy and institutional architecture for this purpose. The changing institutional arrangements around the procurement of engineering assets are the focus of this paper. The paper describes and analyses the transition to new, more collaborative forms of procurement arrangements which are becoming increasingly prevalent in Australia and other OECD countries. Such fundamental shifts from competitive to more collaborative approaches to project governance can be viewed as a major transition in procurement system arrangements. In many ways such changes mirror the shift from New Public Management, with its emphasis on the use of market mechanisms to achieve efficiencies {Hood, 1991 #166}, towards more collaborative approaches to service delivery, such as those under network governance arrangements {Keast, 2007 #925}. However, just as traditional forms of procurement in a market context resulted in unexpected outcomes for industry, such as a fragmented industry afflicted by chronic litigation {Dubois, 2002 #9}, the change to more collaborative forms of procurement is unlikely to be a panacea to the problems of procurement, and may well also have unintended consequences. This paper argues that perspectives from complex adaptive systems (CAS) theory can contribute to the theory and practice of managing system transitions. In particular the concept of emergence provides a key theoretical construct to understand the aggregate effect that individual project governance arrangements can have upon the structure of specific industries, which in turn impact individual projects. Emergence is understood here as the macro structure that emerges out of the interaction of agents in the system {Holland, 1998 #100; Tang, 2006 #51}.

A soft approach to solving hard problems in construction project management

Construction projects are faced with a challenge that must not be underestimated. These projects are increasingly becoming highly competitive, more complex, and difficult to manage. They become ‘wicked problems’, which are difficult to solve using traditional approaches. Soft Systems Methodology (SSM) is a systems approach that is used for analysis and problem solving in such complex and messy situations. SSM uses “systems thinking” in a cycle of action research, learning and reflection to help understand the various perceptions that exist in the minds of the different people involved in the situation. This paper examines the benefits of applying SSM to wicked problems in construction project management, especially those situations that are challenging to understand and difficult to act upon. It includes relevant examples of its use in dealing with the confusing situations that incorporate human, organizational and technical aspects.