Towards a theoretical framework for organizational innovation

Being able to innovate has become a critical capability for many contemporary organizations in an effort to sustain their operations in the long run. However, existing innovation models that attempt to guide organizations emphasize different aspects of innovation (e.g., products, services or business models), different stages of innovation (e.g., ideation, implementation or operation) or different skills (e.g., development or crowdsourcing) that are necessary to innovate, in turn creating isolated pockets of understanding about different aspects of innovation. In order to yield more predictable innovation outcomes organizations need to understand what exactly they need to focus on, what capabilities they need to have and what is necessary in order to take an idea to market. This paper aims at constructing a framework for innovation that contributes to this understanding. We will focus on a number of different stages in the innovation process and highlight different types and levels of organizational, technological, individual and process capabilities required to manage the organizational innovation process. Our work offers a comprehensive conceptualization of innovation as a multi-level process model, and provides a range of implications for further empirical and theoretical examination.

A public asset management framework for Indonesian local governments

The decentralisation reform in Indonesia has mandated the Central Government to transfer some functions and responsibilities to local governments including the transfer of human resources, assets and budgets. Local governments became giant asset holders almost overnight and most were ill prepared to handle these transformations. Assets were transferred without analysing local government need, ability or capability to manage the assets and no local government was provided with an asset management framework. Therefore, the aim of this research is to develop a Public Asset Management Framework for provincial governments in Indonesia, especially for infrastructure and real property assets. This framework will enable provincial governments to develop integrated asset management procedures throughout asset‘s lifecycle. Achieving the research aim means answering the following three research questions; 1) How do provincial governments in Indonesia currently manage their public assets? 2) What factors influence the provincial governments in managing these public assets? 3) How is a Public Asset Management Framework developed that is specific for the Indonesian provincial governments‘ situation? This research applied case studies approach after a literature review; document retrieval, interviews and observations were collated. Data was collected in June 2009 (preliminary data collection) and January to July 2010 in the major eastern Indonesian provinces. Once the public asset management framework was developed, a focus group was used to verify the framework. Results are threefold and indicate that Indonesian provincial governments need to improve the effectiveness and efficiency of current practice of public asset management in order to improve public service quality. The second result shows that the 5 major concerns that influence the local government public asset management processes are asset identification and inventory systems, public asset holding, asset guidance and legal arrangements, asset management efficiency and effectiveness, and, human resources and their organisational arrangements. The framework was then applied to assets already transferred to local governments and so included a system of asset identification and a needs analysis to classify the importance of these assets to local governments, their functions and responsibilities in delivering public services. Assets that support local government functions and responsibilities will then be managed using suitable asset lifecycle processes. Those categorised as surplus assets should be disposed. Additionally functions and responsibilities that do not need an asset solution should be performed directly by local governments. These processes must be measured using performance measurement indicators. All these stages should be guided and regulated with sufficient laws and regulations. Constant improvements to the quality and quantity of human resources hold an important role in successful public asset management processes. This research focuses on developing countries, and contributes toward the knowledge of a Public Asset Management Framework at local government level, particularly Indonesia. The framework provides local governments a foundation to improve their effectiveness and efficiency in managing public assets, which could lead to improved public service quality. This framework will ensure that the best decisions are made throughout asset decision ownership and provide a better asset life cycle process, leading to selection of the most appropriate asset, improve its acquisition and delivery process, optimise asset performance, and provide an appropriate disposal program.

Identification and evaluation of service bundles for governmental one-stop portals

Recent advances in the area of ‘Transformational Government’ position the citizen at the centre of focus. This paradigm shift from a department-centric to a citizen-centric focus requires governments to re-think their approach to service delivery, thereby decreasing costs and increasing citizen satisfaction. The introduction of franchises as a virtual business layer between the departments and their citizens is intended to provide a solution. Franchises are structured to address the needs of citizens independent of internal departmental structures. For delivering services online, governments pursue the development of a One-Stop Portal, which structures information and services through those franchises. Thus, each franchise can be mapped to a specific service bundle, which groups together services that are deemed to be of relevance to a specific citizen need. This study focuses on the development and evaluation of these service bundles. In particular, two research questions guide the line of investigation of this study: Research Question 1): What methods can be used by governments to identify service bundles as part of governmental One-Stop Portals? Research Question 2): How can the quality of service bundles in governmental One-Stop Portals be evaluated? The first research question asks about the identification of suitable service bundle identification methods. A literature review was conducted, to, initially, conceptualise the service bundling task, in general. As a consequence, a 4-layer model of service bundling and a morphological box were created, detailing characteristics that are of relevance when identifying service bundles. Furthermore, a literature review of Decision-Support Systems was conducted to identify approaches of relevance in different bundling scenarios. These initial findings were complemented by targeted studies of multiple leading governments in the e-government domain, as well as with a local expert in the field. Here, the aim was to identify the current status of online service delivery and service bundling in practice. These findings led to the conceptualising of two service bundle identification methods, applicable in the context of Queensland Government: On the one hand, a provider-driven approach, based on service description languages, attributes, and relationships between services was conceptualised. As well, a citizen-driven approach, based on analysing the outcomes from content identification and grouping workshops with citizens, was also conceptualised. Both methods were then applied and evaluated in practice. The conceptualisation of the provider-driven method for service bundling required the initial specification of relevant attributes that could be used to identify similarities between services called relationships; these relationships then formed the basis for the identification of service bundles. This study conceptualised and defined seven relationships, namely ‘Co-location’, ‘Resource’, ‘Co-occurrence’, ‘Event’, ‘Consumer’, ‘Provider’, and ‘Type’. The relationships, and the bundling method itself, were applied and refined as part of six Action Research cycles in collaboration with the Queensland Government. The findings show that attributes and relationships can be used effectively as a means for bundle identification, if distinct decision rules are in place to prescribe how services are to be identified. For the conceptualisation of the citizen-driven method, insights from the case studies led to the decision to involve citizens, through card sorting activities. Based on an initial list of services, relevant for a certain franchise, participating citizens grouped services according to their liking. The card sorting activity, as well as the required analysis and aggregation of the individual card sorting results, was analysed in depth as part of this study. A framework was developed that can be used as a decision-support tool to assist with the decision of what card sorting analysis method should be utilised in a given scenario. The characteristic features associated with card sorting in a government context led to the decision to utilise statistical analysis approaches, such as cluster analysis and factor analysis, to aggregate card sorting results. The second research question asks how the quality of service bundles can be assessed. An extensive literature review was conducted focussing on bundle, portal, and e-service quality. It was found that different studies use different constructs, terminology, and units of analysis, which makes comparing these models a difficult task. As a direct result, a framework was conceptualised, that can be used to position past and future studies in this research domain. Complementing the literature review, interviews conducted as part of the case studies with leaders in e-government, indicated that, typically, satisfaction is evaluated for the overall portal once the portal is online, but quality tests are not conducted during the development phase. Consequently, a research model which appropriately defines perceived service bundle quality would need to be developed from scratch. Based on existing theory, such as Theory of Reasoned Action, Expectation Confirmation Theory, and Theory of Affordances, perceived service bundle quality was defined as an inferential belief. Perceived service bundle quality was positioned within the nomological net of services. Based on the literature analysis on quality, and on the subsequent work of a focus group, the hypothesised antecedents (descriptive beliefs) of the construct and the associated question items were defined and the research model conceptualised. The model was then tested, refined, and finally validated during six Action Research cycles. Results show no significant difference in higher quality or higher satisfaction among users for either the provider-driven method or for the citizen-driven method. The decision on which method to choose, it was found, should be based on contextual factors, such as objectives, resources, and the need for visibility. The constructs of the bundle quality model were examined. While the quality of bundles identified through the citizen-centric approach could be explained through the constructs ‘Navigation’, ‘Ease of Understanding’, and ‘Organisation’, bundles identified through the provider-driven approach could be explained solely through the constructs ‘Navigation’ and ‘Ease of Understanding’. An active labelling style for bundles, as part of the provider-driven Information Architecture, had a larger impact on ‘Quality’ than the topical labelling style used in the citizen-centric Information Architecture. However, ‘Organisation’, reflecting the internal, logical structure of the Information Architecture, was a significant factor impacting on ‘Quality’ only in the citizen-driven Information Architecture. Hence, it was concluded that active labelling can compensate for a lack of logical structure. Further studies are needed to further test this conjecture. Such studies may involve building alternative models and conducting additional empirical research (e.g. use of an active labelling style for the citizen-driven Information Architecture). This thesis contributes to the body of knowledge in several ways. Firstly, it presents an empirically validated model of the factors explaining and predicting a citizen’s perception of service bundle quality. Secondly, it provides two alternative methods that can be used by governments to identify service bundles in structuring the content of a One-Stop Portal. Thirdly, this thesis provides a detailed narrative to suggest how the recent paradigm shift in the public domain, towards a citizen-centric focus, can be pursued by governments; the research methodology followed by this study can serve as an exemplar for governments seeking to achieve a citizen-centric approach to service delivery.

Towards green universities : a decision support framework for green roof and living wall implementation

This research examined why university campus development has not fully embraced green technology despite common expectations. Semi-structured interviews and a Delphi Study explored universities’ organisational issues and delivery processes for projects with a sustainability focus. Critical organisational components and their internal relationships were studied, and critical factors for project success identified. A decision-making framework was developed to provide strategic directions for universities to optimise organisational environment and overcome barriers in order to better deliver sustainable projects on campuses.

An integrated framework for assessing community resilience in disaster management

Climate change is predicted to increase the frequency and severity of extreme weather events which pose significant challenges to the ability of government and other relief agencies to plan for, cope with and respond to disasters. Consequently, it is important that communities in climate sensitive and potential disaster prone areas strengthen their resilience to natural disasters in order to expeditiously recover from potential disruptions and damage caused by disasters. Building self reliance and, particularly in the immediate aftermath of a disaster, can facilitate short-term and long-term community recovery. To build stronger and more resilient communities, it is essential to have a better understanding of their current resilience capabilities by assessing areas of strength, risks and vulnerabilities so that their strengths can be enhanced and the risks and vulnerability can be appropriately addressed and mitigated through capacity building programs. While a number of conceptual frameworks currently exist to assess the resilience level of communities to disasters, they have tended to differ on their emphasis, scope and definition of what constitutes community resilience and how community resilience can be most effectively and accurately assessed. These limitations are attributed to the common approach of viewing community resilience through a mono-disciplinary lens. To overcome this, this paper proposes an integrated conceptual framework that takes into account the complex interplay of environmental, social, governance, infrastructure and economic attributes associated with community resilience. The framework can be operationalised using a range of resilience indicators to suit the nature of a disaster and the specific characteristics of a study region.

Improving statistical analysis of matched case-control studies

Matched case–control research designs can be useful because matching can increase power due to reduced variability between subjects. However, inappropriate statistical analysis of matched data could result in a change in the strength of association between the dependent and independent variables or a change in the significance of the findings. We sought to ascertain whether matched case–control studies published in the nursing literature utilized appropriate statistical analyses. Of 41 articles identified that met the inclusion criteria, 31 (76%) used an inappropriate statistical test for comparing data derived from case subjects and their matched controls. In response to this finding, we developed an algorithm to support decision-making regarding statistical tests for matched case–control studies.

Hydrogeological framework, conceptual and numerical groundwater flow model of Laidley Creek catchment, Queensland, Australia

This thesis studies the water resources of Laidley Creek catchment within the Lockyer Valley where groundwater is used for intensive irrigation of crops. A holistic approach was used to consider groundwater within the total water cycle. The project mapped the geology, measured stream flows and groundwater levels, and analysed the chemistry of the waters. These data were integrated within a catchment-wide conceptual model, including historic and rainfall records. From this a numerical simulation was produced to test data validity and develop predictions of behaviour, which can support management decisions, particularly in times of variable climate.

Developing a sustainability impact assessment framework for local urban planning

In order to meet the land use and infrastructure needs of the community with the additional challenges posed by climate change and a global recession, it is essential that Queensland local governments test their proposed integrated land use and infrastructure plans to ensure the maximum achievement of triple-bottom line sus-tainability goals. Extensive regulatory impact assessment systems are in place at the Australian and state government levels to substantiate and test policy and legislative proposals, however no such requirement has been extended to the local government level. This paper contends that with the devolution of responsibility to local government and growing impacts of local government planning and development assessment activities, impact assessment of regulatory planning instruments is appropriate and overdue. This is particularly so in the Queensland context where local governments manage metropolitan and regional scale responsibilities and their planning schemes under the Sustainable Planning Act 2009 integrate land use and infrastructure planning to direct development rights, the spatial allocation of land, and infrastructure investment. It is critical that urban planners have access to fit-for-purpose impact assessment frameworks which support this challenging task and address the important relationship between local planning and sustainable urban development. This paper uses two examples of sustainability impact assessment and a case study from the Queensland local urban planning context to build an argument and potential starting point for impact assessment in local planning processes.

Valuing student and community voices in the university : action research as a framework for community service-learning

The Community Service-learning Lab (the Lab) was initiated as a university-wide service-learning experience at an Australian university. The Lab engages students, academics, and key community organisations in interdisciplinary action research projects to support student learning and to explore complex and ongoing problems nominated by the community partners. The current study uses feedback from the first offering of the Lab and focuses on exploring student experiences of the service learning project using an action research framework. Student reflections on this experience have revealed some positive outcomes of the Lab such as an appreciation for positive and strengths-based change. These outcomes are corroborated by collected reflections from community partners and academics. The students also identified challenges balancing the requirements for assessment and their goals to serve the community partner’s needs. This feedback has provided vital information for the academic team, highlighting the difficulties in balancing the agenda of the academic framework and the desire to give students authentic experiences.

Framework for implementing design build project delivery system in road infrastructure in Indonesia

The thesis provides a framework for potential implementation of the design-build (DB) project delivery system in road infrastructure projects in Indonesia. This framework proposed a structure of the hierarchy of factors promoting the potential implementation of the DB project delivery system and introduced ways to implement the DB project delivery system through level of hierarchical factors. These findings not only give benefit to the academic knowledge but also to the public officials in guiding them with regard to the priority of promoting factors in the process to implement the DB system.

Post-disaster reconstruction: a statistical overview of the barriers for community participation in Afghanistan

When a community already torn by an event such as a prolonged war, is then hit by a natural disaster, the negative impact of this subsequent disaster in the longer term can be extremely devastating. Natural disasters further damage already destabilised and demoralised communities, making it much harder for them to be resilient and recover. Communities often face enormous challenges during the immediate recovery and the subsequent long term reconstruction periods, mainly due to the lack of a viable community involvement process. In post-war settings, affected communities, including those internally displaced, are often conceived as being completely disabled and are hardly ever consulted when reconstruction projects are being instigated. This lack of community involvement often leads to poor project planning, decreased community support, and an unsustainable completed project. The impact of war, coupled with the tensions created by the uninhabitable and poor housing provision, often hinders the affected residents from integrating permanently into their home communities. This paper outlines a number of fundamental factors that act as barriers to community participation related to natural disasters in post-war settings. The paper is based on a statistical analysis of, and findings from, a questionnaire survey administered in early 2012 in Afghanistan.

A micro-level indexing model for the assessment of sustainable urban ecosystems

During the last several decades, the quality of natural resources and their services have been exposed to significant degradation from increased urban populations combined with the sprawl of settlements, development of transportation networks and industrial activities (Dorsey, 2003; Pauleit et al., 2005). As a result of this environmental degradation, a sustainable framework for urban development is required to provide the resilience of natural resources and ecosystems. Sustainable urban development refers to the management of cities with adequate infrastructure to support the needs of its population for the present and future generations as well as maintain the sustainability of its ecosystems (UNEP/IETC, 2002; Yigitcanlar, 2010). One of the important strategic approaches for planning sustainable cities is „ecological planning‟. Ecological planning is a multi-dimensional concept that aims to preserve biodiversity richness and ecosystem productivity through the sustainable management of natural resources (Barnes et al., 2005). As stated by Baldwin (1985, p.4), ecological planning is the initiation and operation of activities to direct and control the acquisition, transformation, disruption and disposal of resources in a manner capable of sustaining human activities with a minimum disruption of ecosystem processes. Therefore, ecological planning is a powerful method for creating sustainable urban ecosystems. In order to explore the city as an ecosystem and investigate the interaction between the urban ecosystem and human activities, a holistic urban ecosystem sustainability assessment approach is required. Urban ecosystem sustainability assessment serves as a tool that helps policy and decision-makers in improving their actions towards sustainable urban development. There are several methods used in urban ecosystem sustainability assessment among which sustainability indicators and composite indices are the most commonly used tools for assessing the progress towards sustainable land use and urban management. Currently, a variety of composite indices are available to measure the sustainability at the local, national and international levels. However, the main conclusion drawn from the literature review is that they are too broad to be applied to assess local and micro level sustainability and no benchmark value for most of the indicators exists due to limited data availability and non-comparable data across countries. Mayer (2008, p. 280) advocates that by stating "as different as the indices may seem, many of them incorporate the same underlying data because of the small number of available sustainability datasets". Mori and Christodoulou (2011) also argue that this relative evaluation and comparison brings along biased assessments, as data only exists for some entities, which also means excluding many nations from evaluation and comparison. Thus, there is a need for developing an accurate and comprehensive micro-level urban ecosystem sustainability assessment method. In order to develop such a model, it is practical to adopt an approach that uses a method to utilise indicators for collecting data, designate certain threshold values or ranges, perform a comparative sustainability assessment via indices at the micro-level, and aggregate these assessment findings to the local level. Hereby, through this approach and model, it is possible to produce sufficient and reliable data to enable comparison at the local level, and provide useful results to inform the local planning, conservation and development decision-making process to secure sustainable ecosystems and urban futures. To advance research in this area, this study investigated the environmental impacts of an existing urban context by using a composite index with an aim to identify the interaction between urban ecosystems and human activities in the context of environmental sustainability. In this respect, this study developed a new comprehensive urban ecosystem sustainability assessment tool entitled the „Micro-level Urban-ecosystem Sustainability IndeX‟ (MUSIX). The MUSIX model is an indicator-based indexing model that investigates the factors affecting urban sustainability in a local context. The model outputs provide local and micro-level sustainability reporting guidance to help policy-making concerning environmental issues. A multi-method research approach, which is based on both quantitative analysis and qualitative analysis, was employed in the construction of the MUSIX model. First, a qualitative research was conducted through an interpretive and critical literature review in developing a theoretical framework and indicator selection. Afterwards, a quantitative research was conducted through statistical and spatial analyses in data collection, processing and model application. The MUSIX model was tested in four pilot study sites selected from the Gold Coast City, Queensland, Australia. The model results detected the sustainability performance of current urban settings referring to six main issues of urban development: (1) hydrology, (2) ecology, (3) pollution, (4) location, (5) design, and; (6) efficiency. For each category, a set of core indicators was assigned which are intended to: (1) benchmark the current situation, strengths and weaknesses, (2) evaluate the efficiency of implemented plans, and; (3) measure the progress towards sustainable development. While the indicator set of the model provided specific information about the environmental impacts in the area at the parcel scale, the composite index score provided general information about the sustainability of the area at the neighbourhood scale. Finally, in light of the model findings, integrated ecological planning strategies were developed to guide the preparation and assessment of development and local area plans in conjunction with the Gold Coast Planning Scheme, which establishes regulatory provisions to achieve ecological sustainability through the formulation of place codes, development codes, constraint codes and other assessment criteria that provide guidance for best practice development solutions. These relevant strategies can be summarised as follows: • Establishing hydrological conservation through sustainable stormwater management in order to preserve the Earth’s water cycle and aquatic ecosystems; • Providing ecological conservation through sustainable ecosystem management in order to protect biological diversity and maintain the integrity of natural ecosystems; • Improving environmental quality through developing pollution prevention regulations and policies in order to promote high quality water resources, clean air and enhanced ecosystem health; • Creating sustainable mobility and accessibility through designing better local services and walkable neighbourhoods in order to promote safe environments and healthy communities; • Sustainable design of urban environment through climate responsive design in order to increase the efficient use of solar energy to provide thermal comfort, and; • Use of renewable resources through creating efficient communities in order to provide long-term management of natural resources for the sustainability of future generations.

Collaborative learning and critical thinking in technology-enhanced environments : an instructional design framework

The nature and characteristics of how learners learn today are changing. As technology use in learning and teaching continues to grow, its integration to facilitate deep learning and critical thinking becomes a primary consideration. The implications for learner use, implementation strategies, design of integration frameworks and evaluation of their effectiveness in learning environments cannot be overlooked. This study specifically looked at the impact that technology-enhanced learning environments have on different learners’ critical thinking in relation to eductive ability, technological self-efficacy, and approaches to learning and motivation in collaborative groups. These were explored within an instructional design framework called CoLeCTTE (collaborative learning and critical thinking in technology-enhanced environments) which was proposed, revised and used across three cases. The field of investigation was restricted to three key questions: 1) Do learner skill bases (learning approach and eductive ability) influence critical thinking within the proposed CoLeCTTE framework? If so, how?; 2) Do learning technologies influence the facilitation of deep learning and critical thinking within the proposed CoLeCTTE framework? If so, how?; and 3) How might learning be designed to facilitate the acquisition of deep learning and critical thinking within a technology-enabled collaborative environment? The rationale, assumptions and method of research for using a mixed method and naturalistic case study approach are discussed; and three cases are explored and analysed. The study was conducted at the tertiary level (undergraduate and postgraduate) where participants were engaged in critical technical discourse within their own disciplines. Group behaviour was observed and coded, attributes or skill bases were measured, and participants interviewed to acquire deeper insights into their experiences. A progressive case study approach was used, allowing case investigation to be implemented in a "ladder-like" manner. Cases 1 and 2 used the proposed CoLeCTTE framework with more in-depth analysis conducted for Case 2 resulting in a revision of the CoLeCTTE framework. Case 3 used the revised CoLeCTTE framework and in-depth analysis was conducted. The findings led to the final version of the framework. In Cases 1, 2 and 3, content analysis of group work was conducted to determine critical thinking performance. Thus, the researcher used three small groups where learner skill bases of eductive ability, technological self-efficacy, and approaches to learning and motivation were measured. Cases 2 and 3 participants were interviewed and observations provided more in-depth analysis. The main outcome of this study is analysis of the nature of critical thinking within collaborative groups and technology-enhanced environments positioned in a theoretical instructional design framework called CoLeCTTE. The findings of the study revealed the importance of the Achieving Motive dimension of a student’s learning approach and how direct intervention and strategies can positively influence critical thinking performance. The findings also identified factors that can adversely affect critical thinking performance and include poor learning skills, frustration, stress and poor self-confidence, prioritisations over learning; and inadequate appropriation of group role and tasks. These findings are set out as instructional design guidelines for the judicious integration of learning technologies into learning and teaching practice for higher education that will support deep learning and critical thinking in collaborative groups. These guidelines are presented in two key areas: technology and tools; and activity design, monitoring, control and feedback.

Practical framework for measuring performance of international construction firms

The internationalization of construction companies has become of significant interest as the global construction market continues to be integrated into a more competitive and turbulent business environment. However, due to the complicated and multifaceted nature of international business and performance, there is as yet no consensus on how to evaluate the performance of international construction firms (ICFs). The purpose of this paper, therefore, is to develop a practical framework for measuring the performance of ICFs. Based on the balanced scorecard (BSC), a framework with detailed measures is developed, investigated, and tested using a three-step research design. In the first step, 27 measures under six dimensions (financial, market, customer, internal business processes, stakeholders, and learning and growth) are determined by literature review, interviews with academics, and seminar discussions. Subsequently, a questionnaire survey is conducted to investigate weights of these 27 performance measures. The questionnaire survey also supports the importance of measuring intangible aspects of international construction performance from the practitioner’s viewpoint. Additionally, a case study is described to test the framework’s robustness and usefulness. This is achieved by benchmarking the performance of a Chinese ICF with nine other counterparts worldwide. It is found that the framework provides an effective basis for benchmarking ICFs to effectively monitor their performance and support the development of strategies for improved competitiveness in the international arena. This paper is the first attempt to present a balanced and practically tested framework for evaluating the performance of ICFs. It contributes to the practice of performance measurement and related internationalization in the construction industry in general.

An evaluation framework for cross-lingual link discovery

Cross-Lingual Link Discovery (CLLD) is a new problem in Information Retrieval. The aim is to automatically identify meaningful and relevant hypertext links between documents in different languages. This is particularly helpful in knowledge discovery if a multi-lingual knowledge base is sparse in one language or another, or the topical coverage in each language is different; such is the case with Wikipedia. Techniques for identifying new and topically relevant cross-lingual links are a current topic of interest at NTCIR where the CrossLink task has been running since the 2011 NTCIR-9. This paper presents the evaluation framework for benchmarking algorithms for cross-lingual link discovery evaluated in the context of NTCIR-9. This framework includes topics, document collections, assessments, metrics, and a toolkit for pooling, assessment, and evaluation. The assessments are further divided into two separate sets: manual assessments performed by human assessors; and automatic assessments based on links extracted from Wikipedia itself. Using this framework we show that manual assessment is more robust than automatic assessment in the context of cross-lingual link discovery.