Statistical Tests for Joint Analysis of Performance Measures

Recently there has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and architectural complexity). Once one has learned a model based on their devised method, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Unfortunately, the standard tests used for this purpose are not able to jointly consider performance measures. The aim of this paper is to resolve this issue by developing statistical procedures that are able to account for multiple competing measures at the same time. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameter of such models, as usually the number of studied cases is very reduced in such comparisons. Real data from a comparison among general purpose classifiers is used to show a practical application of our tests.

Cloning and Characterization of Two Potent Kunitz Type Protease Inhibitors from Echinococcus granulosus

The tapeworm Echinococcus granulosus is responsible for cystic echinococcosis (CE), a cosmopolitan disease which imposes a significant burden on the health and economy of affected communities. Little is known about the molecular mechanisms whereby E. granulosus is able to survive in the hostile mammalian host environment, avoiding attack by host enzymes and evading immune responses, but protease inhibitors released by the parasite are likely implicated. We identified two nucleotide sequences corresponding to secreted single domain Kunitz type protease inhibitors (EgKIs) in the E. granulosus genome, and their cDNAs were cloned, bacterially expressed and purified. EgKI-1 is highly expressed in the oncosphere (egg) stage and is a potent chymotrypsin and neutrophil elastase inhibitor that binds calcium and reduced neutrophil infiltration in a local inflammation model. EgKI-2 is highly expressed in adult worms and is a potent inhibitor of trypsin. As powerful inhibitors of mammalian intestinal proteases, the EgKIs may play a pivotal protective role in preventing proteolytic enzyme attack thereby ensuring survival of E. granulosus within its mammalian hosts. EgKI-1 may also be involved in the oncosphere in host immune evasion by inhibiting neutrophil elastase and cathepsin G once this stage is exposed to the mammalian blood system. In light of their key roles in protecting E. granulosus from host enzymatic attack, the EgKI proteins represent potential intervention targets to control CE. This is important as new public health measures against CE are required, given the inefficiencies of available drugs and the current difficulties in its treatment and control. In addition, being a small sized highly potent serine protease inhibitor, and an inhibitor of neutrophil chemotaxis, EgKI-1 may have clinical potential as a novel anti-inflammatory therapeutic.

Joint Analysis of Multiple Algorithms and Performance Measures

There has been an increasing interest in the development of new methods using Pareto optimality to deal with multi-objective criteria (for example, accuracy and time complexity). Once one has developed an approach to a problem of interest, the problem is then how to compare it with the state of art. In machine learning, algorithms are typically evaluated by comparing their performance on different data sets by means of statistical tests. Standard tests used for this purpose are able to consider jointly neither performance measures nor multiple competitors at once. The aim of this paper is to resolve these issues by developing statistical procedures that are able to account for multiple competing measures at the same time and to compare multiple algorithms altogether. In particular, we develop two tests: a frequentist procedure based on the generalized likelihood-ratio test and a Bayesian procedure based on a multinomial-Dirichlet conjugate model. We further extend them by discovering conditional independences among measures to reduce the number of parameters of such models, as usually the number of studied cases is very reduced in such comparisons. Data from a comparison among general purpose classifiers is used to show a practical application of our tests.

Testing the psychometric properties of two self-report clinical competence scales for nursing students.

Background: It is important to assess the clinical competence of nursing students to gauge their educational needs. Competence can be measured by self-assessment tools; however, Anema and McCoy (2010) contend that currently available measures should be further psychometrically tested.
Aim: To test the psychometric properties of Nursing Competencies Questionnaire (NCQ) and Self-Efficacy in Clinical Performance (SECP) clinical competence scales.
Method: A non-randomly selected sample of n=248 2nd year nursing students completed NCQ, SECP and demographic questionnaires (June and September 2013). Mokken Scaling Analysis (MSA) was used to investigate structural validity and scale properties; convergent and discriminant validity and reliability were also tested for each scale.
Results: MSA analysis identified that the NCQ is a unidimensional scale with strong scale scalability coefficients Hs =0.581; but limited item rankability HT =0.367. The SECP scale MSA suggested that the scale could be potentially split into two unidimensional scales (SECP28 and SECP7), each with good/reasonable scalablity psychometric properties as summed scales but negligible/very limited scale rankability (SECP28: Hs = 0.55, HT=0.211; SECP7: Hs = 0.61, HT=0.049). Analysis of between cohort differences and NCQ/SECP scores produced evidence of discriminant and convergent validity; good internal reliability was also found: NCQ α = 0.93, SECP28 α = 0.96 and SECP7 α=0.89.

Discussion: In line with previous research further evidence of the NCQ’s reliability and validity was demonstrated. However, as the SECP findings are new and the sample small with reference to Straat and colleagues (2014), the SECP results should be interpreted with caution and verified on a second sample.
Conclusions: Measurement of perceived self-competence could start early in a nursing programme to support students’ development of clinical competence. Further testing of the SECP scale with larger nursing student samples from different programme years is indicated.

References:
Anema, M., G and McCoy, JK. (2010) Competency-Based Nursing Education: Guide to Achieving Outstanding Learner Outcomes. New York: Springer.
Straat, JH., van der Ark, LA and Sijtsma, K. (2014) Minimum Sample Size Requirements for Mokken Scale Analysis Educational and Psychological Measurement 74 (5), 809-822.