Consumer notes clinical indicators : determining inter-rater reliability with multiple users, nominal categories and several cases

A method of determining inter-rater reliability when there are multiple raters, nominal rating categories and several cases is described and applied in the development of an instrument for auditing the ANZCMHN (1995) standards of practice for mental health nursing in New Zealand. Clinical statements (n=41) from the O’Brien et al (2002a, 2003) study, which reflected nursing behaviours contributing to the achievement of the standards of practice, were used to audit consumer files. During two Phases, the clinical indicator statements were refined and rules for judging the achievement of each statement from case note documentation were established. The resultant statements have adequate inter-rater reliability for the assessment of nursing practice with respect to the ANZCMHN (1995) standards of practice.