Qualitative evaluation of KA24 (Knowledge Access 24)

Durbin, J. & Urquhart, C. (2003). Qualitative evaluation of KA24 (Knowledge Access 24). Aberystwyth: Department of Information Studies, University of Wales Aberystwyth. Sponsorship: Knowledge Access 24 (NHS)

1 Executive summary Aims and objectives of evaluation The aim of the project was to examine patterns of usage, to explore reasons for high and low usage to examine how the benefits of use relate to policy objectives for clinical governance, continuing professional education and quality of patient care. The objectives were to: ? re-assess the enablers and barriers to KA24 use. ? assess the impact of the service on patient care (in terms of improved clinical competence, improved governance arrangements, and quality of patient care) Scope of the evaluation Interviewees were identified through the online survey carried out in May 2003, and the 65 interviewees (56 users, 9 library managers) represented a broad spectrum of users (regular, infrequent, first-time, non-users and library service support). A questionnaire survey was intended to complement the interviews, and this was partially successful, although only 296 out of 700 (response rate 42.3%) forms were returned in time for analysis. The analysis is set in the context of both the First Year Evaluation Report and also the main findings of the online survey conducted in May 2003 by the KA24 team. Key messages Publicity and registration Library staff appear to be informing potential users of the service, and information sheets in the library are also used. Supplemental advertising such as flyers, or posters were mentioned (though rarely) ? there may be a subconscious registering of information in a payslip but it seems likely that general campaigns such as putting information in payslips has to be accompanied by more direct promotion. Most libraries saw only one new user a day, which implies that clinical champions or contacts need to be used to reach more potential users. Publicity campaigns should (if they are not already doing so) focus on departments where there is likely to be more intensive usage of KA24. Champions there might take responsibility for ensuring other colleagues know about the service and get registered. Examples of possible enthusiast departments identified in the survey include: ? practice development units ? clinical audit ? research projects (including systematic review projects) / research nurses ? clinical governance ? medicines information/pharmacies Registration poses a few problems. If the service wishes to profile usage via password use patterns, then more effort needs to go into ensuring that individual users are registered, and that they can re-register or transfer their registration when they move to another hospital within the WDC. This would avoid the fairly common practice of password sharing, when registered users allow other staff access to the service on the registered user?s password. Expectations of the service If the name of the service should reflect the expectations that most registered users have of the service, then the name should reflect ?access to full text journal articles? as that is the main attraction, with databases secondary. If the service decides that a new image and new name is desirable then we recommend that the name should convey both the ?feel? of the service as well as what it is offering. For example, KA24 does convey the message that ?24 hour access? is a key feature and the name is rather like a road name which also conveys the message that this service is about getting you places where you need to go. The key benefit now for users of the KA24/HILO service is access to journal articles, with database access second. The key problem at present is managing unrealistic expectations. Ideally, any new name for the service needs to get both those messages across ? the service is great for getting journal articles, but you can?t ?have it all?. Experience of the service Experience of the service is largely unproblematic. The bad experiences are those associated with dealing with the main expectation of the service ? obtaining full text. Timing out was mentioned as a difficulty for some daily users, for the less frequent users failing to obtain the full text journal articles they think should be there may be viewed as a technical problem. For most, however, the time savings are a key feature of the service, and even those who (apparently) search infrequently appreciated the fact they could search from home, or from their desks. The questionnaire data also indicated that few feel the service is difficult to use, although the strength of opinion on other aspects also suggested that they have insufficient experience of using the service to judge properly. The May online survey suggests that the majority of users are using the service between 2 and 6 times a month. However, the responses to that survey were dominated by acute trusts, and there may be pockets of problems among mental health trusts, primary care trusts and ambulance trusts as responses from those types of trusts were much lower. Clinical benefits realisation There is some evidence of changes to routine practice to incorporate use of KA24 as an integral part of routine clinical or professional practice, but only among the daily users, and particularly among the research active. The questionnaire data confirmed those trends, with over one in four using KA24 for research related purposes, and one in four (not necessarily the same group) frequently using KA24 in work. KA24 is used for a variety of reasons ? CPD and coursework, clinical guideline development, clinical governance activities, general updating and nearly one in five of the questionnaire respondents reported that they occasionally use KA24 for direct patient care queries. The findings from the May online survey reflect that pattern. To provide more benefits, KA24 enhancements would need to make searching easier and faster for clinical, management and research purposes (more cross database searching, seamless linking to full text articles across databases). As indicated earlier, for most users the key attraction is access for full-text articles. Researchers coming from an HE background compare, sometimes unfavourably, the KA24/HILO journal collections with those available in HE. More psychiatry and psychology journals would please mental health and primary care staff. Working patterns and access to the Internet The number of regular Internet users across the interviewees was startling, and confirmed by the questionnaire data which indicated that nearly three in four of the questionnaire survey respondents used the Internet at least once a week. KA24 is likely, therefore, to be viewed as a search engine, although a special type of search engine. Nearly two thirds of the questionnaire respondents could access the Internet from home, and nearly half from their own PC at work. A third had access to a shared PC, or a public PC at work. Only 6% reported problems in access to a PC at work, although some staff in mental health trusts reported they had no option but to use the KA24 service from home. The May online survey findings also reflect the importance of home-based access. There was some evidence that users were beginning to shift towards more reliance on KA24 and less on paper-based resources ? more so for journals than for books. Several of the more active users made suggestions on how to improve KA24 so that it become more tightly integrated into their way of working and researching. Customisation, and integration into other software they are using were the features mentioned. Training and support Those who had received training from the library had found this useful, but a large proportion of the interviewees did not see the need for formal training, particularly when this is difficult to undertake with the variable nature of their clinical workload. Nearly one in ten of the questionnaire respondents did not see the need for training at all. Around half the questionnaire respondents had received some sort of the training, usually informal training from the library on an ?as and when? basis. Interestingly, and probably a reflection of the amount of searching done at home, questionnaire respondents used the online guides within the databases as much as advice from colleagues. Working through tutorials was not as popular as specific advice when required, presumably. Adding up the figures from the questionnaire survey suggests that there is a training gap. If 50% have received some training, 10% are not interested at all, and around 10-20% are making use of online guidance or colleagues? advice, then that leaves 20-30% (probably nurses, particularly in community and primary care) who might appreciate some training. As one of the main advantages to the users of using KA24 is saving time, over other means of obtaining the same information, several expressed interest in some assistance or advice on ?shortcuts? or more advanced features they could use to save time and obtain a better focused search. Libraries might promote ?help-desk? support more, as that would give them better insight into the routine problems (and successes) experienced in their locality. There are many similarities across all the groups (daily users, first time, infrequent, and non-users) in general feelings of competence and confidence in searching. It cannot be assumed, therefore, that a first-time user is a novice searcher, nor that infrequent users see themselves as such. It might be more appropriate to target training and support according to the type of activities or purposes, rather than the competence and confidence of the users. For example, several groups of users could be identified from the interview data: ? A ) Novices, possibly using KA24 to help in coursework, with less confidence using computers (often nurses) ? B) Internet-savvy, using KA24 for coursework and also some research at work (often allied health professionals) ? C) Research active/researchers/ audit or practice development, using KA24 on a regular basis for work, likely to have had (or to have) experience of HE facilities ? D) Infrequent or occasional users who use KA24 for clinical practice or personal research interest as a supportive activity, and may use other resources (could include GPs) ? E) Information professionals (including medicines information and library staff) ? F) Junior doctor ?bright sparks? (often have good ideas for service enhancements).