Overview of the ShARe/CLEF eHealth Evaluation Lab 2014

This paper reports on the 2nd ShARe/CLEFeHealth evaluation lab which continues our evaluation resource building activities for the medical domain. In this lab we focus on patients' information needs as opposed to the more common campaign focus of the specialised information needs of physicians and other healthcare workers. The usage scenario of the lab is to ease patients and next-of-kins' ease in understanding eHealth information, in particular clinical reports. The 1st ShARe/CLEFeHealth evaluation lab was held in 2013. This lab consisted of three tasks. Task 1 focused on named entity recognition and normalization of disorders; Task 2 on normalization of acronyms/abbreviations; and Task 3 on information retrieval to address questions patients may have when reading clinical reports. This year's lab introduces a new challenge in Task 1 on visual-interactive search and exploration of eHealth data. Its aim is to help patients (or their next-of-kin) in readability issues related to their hospital discharge documents and related information search on the Internet. Task 2 then continues the information extraction work of the 2013 lab, specifically focusing on disorder attribute identification and normalization from clinical text. Finally, this year's Task 3 further extends the 2013 information retrieval task, by cleaning the 2013 document collection and introducing a new query generation method and multilingual queries. De-identified clinical reports used by the three tasks were from US intensive care and originated from the MIMIC II database. Other text documents for Tasks 1 and 3 were from the Internet and originated from the Khresmoi project. Task 2 annotations originated from the ShARe annotations. For Tasks 1 and 3, new annotations, queries, and relevance assessments were created. 50, 79, and 91 people registered their interest in Tasks 1, 2, and 3, respectively. 24 unique teams participated with 1, 10, and 14 teams in Tasks 1, 2 and 3, respectively. The teams were from Africa, Asia, Canada, Europe, and North America. The Task 1 submission, reviewed by 5 expert peers, related to the task evaluation category of Effective use of interaction and targeted the needs of both expert and novice users. The best system had an Accuracy of 0.868 in Task 2a, an F1-score of 0.576 in Task 2b, and Precision at 10 (P@10) of 0.756 in Task 3. The results demonstrate the substantial community interest and capabilities of these systems in making clinical reports easier to understand for patients. The organisers have made data and tools available for future research and development.

A critical evaluation of written discharge advice for people with mild traumatic brain injury : what should we be looking for?

Objective: To formally evaluate the written discharge advice for people with mild traumatic brain injury (mTBI). Methods: Eleven publications met the inclusion criteria: (1) intended for adults; (2) ≤two A4 pages; (3) published in English; (4) freely accessible; and (5) currently used (or suitable for use) in Australian hospital emergency departments or similar settings. Two independent raters evaluated the content and style of each publication against established standards. The readability of the publication, the diagnostic term(s) contained in it and a modified Patient Literature Usefulness Index (mPLUI) were also evaluated. Results: The mean content score was 19.18 ± 8.53 (maximum = 31) and the mean style score was 6.8 ± 1.34 (maximum = 8). The mean Flesch-Kincaid reading ease score was 66.42 ± 4.3. The mean mPLUI score was 65.86 ± 14.97 (maximum = 100). Higher scores on these metrics indicate more desirable properties. Over 80% of the publications used mixed diagnostic terminology. One publication scored optimally on two of the four metrics and highly on the others. Discussion: The content, style, readability and usefulness of written mTBI discharge advice was highly variable. The provision of written information to patients with mTBI is advised, but this variability in materials highlights the need for evaluation before distribution. Areas are identified to guide the improvement of written mTBI discharge advice.

Design of a tangible data visualization (extended)

In this paper we describe the design of DNA Jewelry, which is a wearable tangible data representation of personal DNA profile data. An iterative design process was followed to develop a 3D form-language that could be mapped to standard DNA profile data, with the aim of retaining readability of data while also producing an aesthetically pleasing and unique result in the area of personalised design. The work explores design issues with the production of data tangibles, contributes to a growing body of research exploring tangible representations of data and highlights the importance of approaches that move between technology, art and design.