Increasing the rigour and trustworthiness of participatory evaluations : learnings from the field

Participatory evaluation and participatory action research (PAR) are increasingly used in community-based programs and initiatives and there is a growing acknowledgement of their value. These methodologies focus more on knowledge generated and constructed through lived experience than through social science (Vanderplaat 1995). The scientific ideal of objectivity is usually rejected in favour of a holistic approach that acknowledges and takes into account the diverse perspectives, values and interpretations of participants and evaluation professionals. However, evaluation rigour need not be lost in this approach. Increasing the rigour and trustworthiness of participatory evaluations and PAR increases the likelihood that results are seen as credible and are used to continually improve programs and policies.----- Drawing on learnings and critical reflections about the use of feminist and participatory forms of evaluation and PAR over a 10-year period, significant sources of rigour identified include:----- • participation and communication methods that develop relations of mutual trust and open communication----- • using multiple theories and methodologies, multiple sources of data, and multiple methods of data collection----- • ongoing meta-evaluation and critical reflection----- • critically assessing the intended and unintended impacts of evaluations, using relevant theoretical models----- • using rigorous data analysis and reporting processes----- • participant reviews of evaluation case studies, impact assessments and reports.

A time-variant medical data trustworthiness assessment model

Electronic Health Record (EHR) systems are being introduced to overcome the limitations associated with paper-based and isolated Electronic Medical Record (EMR) systems. This is accomplished by aggregating medical data and consolidating them in one digital repository. Though an EHR system provides obvious functional benefits, there is a growing concern about the privacy and reliability (trustworthiness) of Electronic Health Records. Security requirements such as confidentiality, integrity, and availability can be satisfied by traditional hard security mechanisms. However, measuring data trustworthiness from the perspective of data entry is an issue that cannot be solved with traditional mechanisms, especially since degrees of trust change over time. In this paper, we introduce a Time-variant Medical Data Trustworthiness (TMDT) assessment model to evaluate the trustworthiness of medical data by evaluating the trustworthiness of its sources, namely the healthcare organisation where the data was created and the medical practitioner who diagnosed the patient and authorised entry of this data into the patient’s medical record, with respect to a certain period of time. The result can then be used by the EHR system to manipulate health record metadata to alert medical practitioners relying on the information to possible reliability problems.

Ties that blind? How strong ties affect small business owner-managers’ perceived trustworthiness of their advisors

This research investigates how a strong personal relationship (strong tie) between a small business owner-manager and his professional or informal advisor affects the relationship between the advisor's recent performance and the owner-manager's perceptions of the advisor's trustworthiness in terms of ability, benevolence and integrity. A negative moderating effect could point to a 'tie that blinds': the owner-manager may be less critical in evaluating the advisor's perceived trustworthiness in light of their recent performance, because of the existing personal relationship. A conceptual model is constructed and examined with survey data comprising 153 young Finnish businesses. The results show that strong ties increase the owner-manager's perception of the advisor's integrity, disregarding their recent performance. For professional advisors, strong ties reduce the impact of recent performance in the owner-manager's evaluation of their ability. For informal advisors, a strong tie makes it more likely that their benevolence will be evaluated highly in light of their recent performance. While the results show that 'ties can blind' under certain circumstances, the limitations of the study raise the need for further research to specify these contextual factors and examine the causal link between the choice of advisor and business performance.

Interactions between organizational culture, trustworthiness, and mechanisms for inter-project knowledge sharing

This research used a multiple-case study approach to empirically investigate the complex relationship between factors influencing inter-project knowledge sharing—trustworthiness, organizational culture, and knowledge-sharing mechanisms. Adopting a competing values framework, we found evidence of patterns existing between the type of culture, on the project management unit level, and project managers’ perceptions of valuing trustworthy behaviors and the way they share knowledge, on the individual level. We also found evidence for mutually reinforcing the effect of trust and clan culture, which shape tacit knowledge-sharing behaviors.

Using qualitative mixed methods to study small health care organizations while maximising trustworthiness and authenticity

Background The primary health care sector delivers the majority of health care in western countries through small, community-based organizations. However, research into these healthcare organizations is limited by the time constraints and pressure facing them, and the concern by staff that research is peripheral to their work. We developed Q-RARA—Qualitative Rapid Appraisal, Rigorous Analysis—to study small, primary health care organizations in a way that is efficient, acceptable to participants and methodologically rigorous. Methods Q-RARA comprises a site visit, semi-structured interviews, structured and unstructured observations, photographs, floor plans, and social scanning data. Data were collected over the course of one day per site and the qualitative analysis was integrated and iterative. Results We found Q-RARA to be acceptable to participants and effective in collecting data on organizational function in multiple sites without disrupting the practice, while maintaining a balance between speed and trustworthiness. Conclusions The Q-RARA approach is capable of providing a richly textured, rigorous understanding of the processes of the primary care practice while also allowing researchers to develop an organizational perspective. For these reasons the approach is recommended for use in small-scale organizations both within and outside the primary health care sector.

Making visible the coding process : Using qualitative data software in a post-structural study

Qualitative research methods require transparency to ensure the ‘trustworthiness’ of the data analysis. The intricate processes of organizing, coding and analyzing the data are often rendered invisible in the presentation of the research findings, which requires a ‘leap of faith’ for the reader. Computer assisted data analysis software can be used to make the research process more transparent, without sacrificing rich, interpretive analysis by the researcher. This article describes in detail how one software package was used in a poststructural study to link and code multiple forms of data to four research questions for fine-grained analysis. This description will be useful for researchers seeking to use qualitative data analysis software as an analytic tool.

Trust and strong ties in the selection and evaluation of business advisors by small firms

Principal Topic A small firm is unlikely to possess internally the full range of knowledge and skills that it requires or could benefit from for the development of its business. The ability to acquire suitable external expertise - defined as knowledge or competence that is rare in the firm and acquired from the outside - when needed thus becomes a competitive factor in itself. Access to external expertise enables the firm to focus on its core competencies and removes the necessity to internalize every skill and competence. However, research on how small firms access external expertise is still scarce. The present study contributes to this under-developed discussion by analysing the role of trust and strong ties in the small firm's selection and evaluation of sources of external expertise (henceforth referred to as the 'business advisor' or 'advisor'). Granovetter (1973, 1361) defines the strength of a network tie as 'a (probably linear) combination of the amount of time, the emotional intensity, the intimacy (mutual confiding) and the reciprocal services which characterize the tie'. Strong ties in the context of the present investigation refer to sources of external expertise who are well known to the owner-manager, and who may be either informal (e.g., family, friends) or professional advisors (e.g., consultants, enterprise support officers, accountants or solicitors). Previous research has suggested that strong and weak ties have different fortes and the choice of business advisors could thus be critical to business performance) While previous research results suggest that small businesses favour previously well known business advisors, prior studies have also pointed out that an excessive reliance on a network of well known actors might hamper business development, as the range of expertise available through strong ties is limited. But are owner-managers of small businesses aware of this limitation and does it matter to them? Or does working with a well-known advisor compensate for it? Hence, our research model first examines the impact of the strength of tie on the business advisor's perceived performance. Next, we ask what encourages a small business owner-manager to seek advice from a strong tie. A recent exploratory study by Welter and Kautonen (2005) drew attention to the central role of trust in this context. However, while their study found support for the general proposition that trust plays an important role in the choice of advisors, how trust and its different dimensions actually affect this choice remained ambiguous. The present paper develops this discussion by considering the impact of the different dimensions of perceived trustworthiness, defined as benevolence, integrity and ability, on the strength of tie. Further, we suggest that the dimensions of perceived trustworthiness relevant in the choice of a strong tie vary between professional and informal advisors. Methodology/Key Propositions Our propositions are examined empirically based on survey data comprising 153 Finnish small businesses. The data are analysed utilizing the partial least squares (PLS) approach to structural equation modelling with SmartPLS 2.0. Being non-parametric, the PLS algorithm is particularly well-suited to analysing small datasets with non-normally distributed variables. Results and Implications The path model shows that the stronger the tie, the more positively the advisor's performance is perceived. Hypothesis 1, that strong ties will be associated with higher perceptions of performance is clearly supported. Benevolence is clearly the most significant predictor of the choice of a strong tie for external expertise. While ability also reaches a moderate level of statistical significance, integrity does not have a statistically significant impact on the choice of a strong tie. Hence, we found support for two out of three independent variables included in Hypothesis 2. Path coefficients differed between the professional and informal advisor subsamples. The results of the exploratory group comparison show that Hypothesis 3a regarding ability being associated with strong ties more pronouncedly when choosing a professional advisor was not supported. Hypothesis 3b arguing that benevolence is more strongly associated with strong ties in the context of choosing an informal advisor received some support because the path coefficient in the informal advisor subsample was much larger than in the professional advisor subsample. Hypothesis 3c postulating that integrity would be more strongly associated with strong ties in the choice of a professional advisor was supported. Integrity is the most important dimension of trustworthiness in this context. However, integrity is of no concern, or even negative, when using strong ties to choose an informal advisor. The findings of this study have practical relevance to the enterprise support community. First of all, given that the strength of tie has a significant positive impact on the advisor's perceived performance, this implies that small business owners appreciate working with advisors in long-term relationships. Therefore, advisors are well advised to invest into relationship building and maintenance in their work with small firms. Secondly, the results show that, especially in the context of professional advisors, the advisor's perceived integrity and benevolence weigh more than ability. This again emphasizes the need to invest time and effort into building a personal relationship with the owner-manager, rather than merely maintaining a professional image and credentials. Finally, this study demonstrates that the dimensions of perceived trustworthiness are orthogonal with different effects on the strength of tie and ultimately perceived performance. This means that entrepreneurs and advisors should consider the specific dimensions of ability, benevolence and integrity, rather than rely on general perceptions of trustworthiness in their advice relationships.

A medical data reliability assessment model

There is currently a strong focus worldwide on the potential of large-scale Electronic Health Record (EHR) systems to cut costs and improve patient outcomes through increased efficiency. This is accomplished by aggregating medical data from isolated Electronic Medical Record databases maintained by different healthcare providers. Concerns about the privacy and reliability of Electronic Health Records are crucial to healthcare service consumers. Traditional security mechanisms are designed to satisfy confidentiality, integrity, and availability requirements, but they fail to provide a measurement tool for data reliability from a data entry perspective. In this paper, we introduce a Medical Data Reliability Assessment (MDRA) service model to assess the reliability of medical data by evaluating the trustworthiness of its sources, usually the healthcare provider which created the data and the medical practitioner who diagnosed the patient and authorised entry of this data into the patient’s medical record. The result is then expressed by manipulating health record metadata to alert medical practitioners relying on the information to possible reliability problems.

A rights management approach to securing data distribution in coalitions

As network capacity has increased over the past decade, individuals and organisations have found it increasingly appealing to make use of remote services in the form of service-oriented architectures and cloud computing services. Data processed by remote services, however, is no longer under the direct control of the individual or organisation that provided the data, leaving data owners at risk of data theft or misuse. This paper describes a model by which data owners can control the distribution and use of their data throughout a dynamic coalition of service providers using digital rights management technology. Our model allows a data owner to establish the trustworthiness of every member of a coalition employed to process data, and to communicate a machine-enforceable usage policy to every such member.

Privacy and trust management for electronic health records

Establishing a nationwide Electronic Health Record system has become a primary objective for many countries around the world, including Australia, in order to improve the quality of healthcare while at the same time decreasing its cost. Doing so will require federating the large number of patient data repositories currently in use throughout the country. However, implementation of EHR systems is being hindered by several obstacles, among them concerns about data privacy and trustworthiness. Current IT solutions fail to satisfy patients’ privacy desires and do not provide a trustworthiness measure for medical data. This thesis starts with the observation that existing EHR system proposals suer from six serious shortcomings that aect patients’ privacy and safety, and medical practitioners’ trust in EHR data: accuracy and privacy concerns over linking patients’ existing medical records; the inability of patients to have control over who accesses their private data; the inability to protect against inferences about patients’ sensitive data; the lack of a mechanism for evaluating the trustworthiness of medical data; and the failure of current healthcare workflow processes to capture and enforce patient’s privacy desires. Following an action research method, this thesis addresses the above shortcomings by firstly proposing an architecture for linking electronic medical records in an accurate and private way where patients are given control over what information can be revealed about them. This is accomplished by extending the structure and protocols introduced in federated identity management to link a patient’s EHR to his existing medical records by using pseudonym identifiers. Secondly, a privacy-aware access control model is developed to satisfy patients’ privacy requirements. The model is developed by integrating three standard access control models in a way that gives patients access control over their private data and ensures that legitimate uses of EHRs are not hindered. Thirdly, a probabilistic approach for detecting and restricting inference channels resulting from publicly-available medical data is developed to guard against indirect accesses to a patient’s private data. This approach is based upon a Bayesian network and the causal probabilistic relations that exist between medical data fields. The resulting definitions and algorithms show how an inference channel can be detected and restricted to satisfy patients’ expressed privacy goals. Fourthly, a medical data trustworthiness assessment model is developed to evaluate the quality of medical data by assessing the trustworthiness of its sources (e.g. a healthcare provider or medical practitioner). In this model, Beta and Dirichlet reputation systems are used to collect reputation scores about medical data sources and these are used to compute the trustworthiness of medical data via subjective logic. Finally, an extension is made to healthcare workflow management processes to capture and enforce patients’ privacy policies. This is accomplished by developing a conceptual model that introduces new workflow notions to make the workflow management system aware of a patient’s privacy requirements. These extensions are then implemented in the YAWL workflow management system.

Celebrity entrepreneurship and celebrity endorsement : similarities, differences and the effect of deeper engagement

Increasingly, celebrities appear not only as endorsers for products but are apparently engaged in entrepreneurial roles as initiators, owners and perhaps even managers in the ventures that market the products they promote. Despite being extensively referred to in popular media, scholars have been slow to recognise the importance of this new phenomenon. This thesis argues theoretically and shows empirically that celebrity entrepreneurs are more effective communicators than typical celebrity endorsers because of their increased engagement with ventures. I theorise that greater engagement increases the celebrity‘s emotional involvement as perceived by consumers. This is an endorser quality thus far neglected in the marketing communications literature. In turn, emotional involvement, much like the empirically established dimensions trustworthiness, expertise and attractiveness, should affect traditional outcome variables such as attitude towards the advertisement and brand. On the downside, increases in celebrity engagement may lead to relatively stronger and worsening changes in attitudes towards the brand if and when negative information about the celebrity is revealed. A series of eight experiments was conducted on 781 Swedish and Baltic students and 151 Swedish retirees. Though there were nuanced differences and additional complexities in each experiment, participants‘ reactions to advertisements containing a celebrity portrayed as a typical endorser or entrepreneur were recorded. The overall results of these experiments suggest that emotional involvement can be successfully operationalised as distinct from variables previously known to influence communication effectiveness. In addition, emotional involvement has positive effects on attitudes toward the advertisement and brand that are as strong as the predictors traditionally applied in the marketing communications literature. Moreover, the celebrity entrepreneur condition in the experimental manipulation consistently led to an increase in emotional involvement and to a lesser extent trustworthiness, but not expertise and attractiveness. Finally, negative celebrity information led to a change in participants‘ attitudes towards the brand which were more strongly negative for celebrity entrepreneurs than celebrity endorsers. In addition, the effect of negative celebrity information on a company‘s brand is worse when they support the celebrity rather than fire them. However, this effect did not appear to interact with the celebrity‘s purported engagement.

Content and functionality of alcohol and other drug websites : results of an online survey

Background: There is a growing trend for individuals to seek health information from online sources. Alcohol and other drug (AOD) use is a significant health problem worldwide, but access and use of AOD websites is poorly understood. ----- ----- Objective: To investigate content and functionality preferences for AOD and other health websites. Methods: An anonymous online survey examined general Internet and AOD-specific usage and search behaviors, valued features of AOD and health-related websites (general and interactive website features), indicators of website trustworthiness, valued AOD website tools or functions, and treatment modality preferences. ----- ----- Results: Surveys were obtained from 1214 drug (n = 766) and alcohol website users (n = 448) (mean age 26.2 years, range 16-70). There were no significant differences between alcohol and drug groups on demographic variables, Internet usage, indicators of website trustworthiness, or on preferences for AOD website functionality. A robust website design/navigation, open access, and validated content provision were highly valued by both groups. While attractiveness and pictures or graphics were also valued, high-cost features (videos, animations, games) were minority preferences. Almost half of respondents in both groups were unable to readily access the information they sought. Alcohol website users placed greater importance on several AOD website tools and functions than did those accessing other drug websites: online screening tools (χ²2 = 15.8, P < .001, n = 985); prevention programs (χ²2 = 27.5, P < .001, n = 981); tracking functions (χ²2 = 11.5, P = .003, n = 983); self help treatment programs (χ²2 = 8.3, P = .02, n = 984); downloadable fact sheets for friends (χ²2 = 11.6, P = .003, n = 981); or family (χ²2 = 12.7, P = .002, n = 983). The most preferred online treatment option for both the user groups was an Internet site with email therapist support. Explorations of demographic differences were also performed. While gender did not affect survey responses, younger respondents were more likely to value interactive and social networking features, whereas downloading of credible information was most highly valued by older respondents. ----- ----- Conclusions: Significant deficiencies in the provision of accessible information on AOD websites were identified, an important problem since information seeking was the most common reason for accessing these websites, and, therefore, may be a key avenue for engaging website users in behaviour change. The few differences between AOD website users suggested that both types of websites may have similar features, although alcohol website users may more readily be engaged in screening, prevention and self-help programs, tracking change, and may value fact sheets more highly. While the sociodemographic differences require replication and clarification, these differences support the notion that the design and features of AOD websites should target specific audiences to have maximal impact.

High-Order Concept Associations Mining and Inferential Language Modeling for Online Review Spam Detection

Despite many incidents about fake online consumer reviews have been reported, very few studies have been conducted to date to examine the trustworthiness of online consumer reviews. One of the reasons is the lack of an effective computational method to separate the untruthful reviews (i.e., spam) from the legitimate ones (i.e., ham) given the fact that prominent spam features are often missing in online reviews. The main contribution of our research work is the development of a novel review spam detection method which is underpinned by an unsupervised inferential language modeling framework. Another contribution of this work is the development of a high-order concept association mining method which provides the essential term association knowledge to bootstrap the performance for untruthful review detection. Our experimental results confirm that the proposed inferential language model equipped with high-order concept association knowledge is effective in untruthful review detection when compared with other baseline methods.

An integrated framework to assess financial reward systems in construction projects

Motivation is a major driver of project performance. Despite team member ability to deliver successful project outcomes if they are not positively motivated to pursue joint project goals, then performance will be constrained. One approach to improving the motivation of project organizations is by offering a financial reward for the achievement of set performance standards above a minimum required level. However, little investigation has been undertaken into the features of successful incentive systems as a part of an overall delivery strategy. With input from organizational management literature, and drawing on the literature covering psychological and economic theories of motivation, this paper presents an integrated framework that can be used by project organizations to assess the impact of financial reward systems on motivation in construction projects. The integrated framework offers four motivation indicators which reflect key theoretical concepts across both psychological and economic disciplines. The indicators are: (1) Goal Commitment, (2) Distributive Justice, (3) Procedural Justice, and (4) Reciprocity. The paper also interprets the integrated framework against the results of a successful Australian social infrastructure project case study and identifies key learning’s for project organizations to consider when designing financial reward systems. Case study results suggest that motivation directed towards the achievement of incentive goals is influenced not only by the value placed on the financial reward for commercial benefit, but also driven by the strength of the project initiatives that encourage just and fair dealings, supporting the establishment of trust and positive reciprocal behavior across a project team. The strength of the project relationships was found to be influenced by how attractive the achievement of the goal is to the incentive recipient and how likely they were to push for the achievement of the goal. Interestingly, findings also suggested that contractor motivation is also influenced by the fairness of the performance measurement process and their perception of the trustworthiness and transparency of their client. These findings provide the basis for future research on the impact of financial reward systems on motivation in construction projects. It is anticipated that such research will shed new light on this complex topic and further define how reward systems should be designed to promote project team motivation. Due to the unique nature of construction projects with high levels of task complexity and interdependence, results are expected to vary in comparison to previous studies based on individuals or single-entity organizations.