Inferential, non-parametric statistics to assess quality of probabilistic forecast systems

Many statistical forecast systems are available to interested users. In order to be useful for decision-making, these systems must be based on evidence of underlying mechanisms. Once causal connections between the mechanism and their statistical manifestation have been firmly established, the forecasts must also provide some quantitative evidence of `quality’. However, the quality of statistical climate forecast systems (forecast quality) is an ill-defined and frequently misunderstood property. Often, providers and users of such forecast systems are unclear about what ‘quality’ entails and how to measure it, leading to confusion and misinformation. Here we present a generic framework to quantify aspects of forecast quality using an inferential approach to calculate nominal significance levels (p-values) that can be obtained either by directly applying non-parametric statistical tests such as Kruskal-Wallis (KW) or Kolmogorov-Smirnov (KS) or by using Monte-Carlo methods (in the case of forecast skill scores). Once converted to p-values, these forecast quality measures provide a means to objectively evaluate and compare temporal and spatial patterns of forecast quality across datasets and forecast systems. Our analysis demonstrates the importance of providing p-values rather than adopting some arbitrarily chosen significance levels such as p < 0.05 or p < 0.01, which is still common practice. This is illustrated by applying non-parametric tests (such as KW and KS) and skill scoring methods (LEPS and RPSS) to the 5-phase Southern Oscillation Index classification system using historical rainfall data from Australia, The Republic of South Africa and India. The selection of quality measures is solely based on their common use and does not constitute endorsement. We found that non-parametric statistical tests can be adequate proxies for skill measures such as LEPS or RPSS. The framework can be implemented anywhere, regardless of dataset, forecast system or quality measure. Eventually such inferential evidence should be complimented by descriptive statistical methods in order to fully assist in operational risk management.

Assessing performance in critical care nursing practice using the Benner Novice to expert model: A pilot study

Background It is often believed that by ensuring the ongoing completion of competency documents and life-long learning in nursing practice guarantees quality patient care. This is probably true in most cases where it provides reassurances that the nursing team is maintaining a safe “generalised” level of practice. However, competency does not always promise quality performance. There are a number of studies that have reported differences in what practitioners know and what they actually do despite being deemed competent. Aim The aim of this study was to assess whether our current competency documentation is fit for purpose and to ascertain whether performance assessment needs to be a key component in determining competence. Method 15 nurses within a General ICU who had been on the unit <4 years agreed to participate in this project. Using participant observation and assessing performance against key indicators of the Benner Novice to Expert5 model the participants were supported and assessed over the course of a ‘normal’ nursing shift. Results The results were surprising both positively and negatively. First, the nurses felt more empowered in their clinical decision making skills; second, it identified individual learning needs and milestones in educational development. There were some key challenges identified which included 5 nurses over estimating their level of competence, practice was still very much focused on task acquisition and skill and surprisingly some nurses still felt dominated by the other health professionals within the unit. Conclusion We found that the capacity and capabilities of our nursing workforce needs continual ongoing support especially if we want to move our staff from capable task-doer to competent performers. Using the key novice to expert indicators identified the way forward for us in how we assess performance and competence in practice particularly where promotion to higher grades is based on existing documentation.

Assessment of transit quality of service with occupancy load factor and passenger travel time measures

This paper investigates quality of service (QoS) and resource productivity implications of transit route passenger loading and travel time. It highlights the value of occupancy load factor as a direct passenger comfort QoS measure. Automatic Fare Collection data for a premium radial bus route in Brisbane, Australia, is used to investigate time series correlation between occupancy load factor and passenger average travel time. Correlation is strong across the entire span of service in both directions. Passengers tend to be making longer, peak direction commuter trips under significantly less comfortable conditions than off-peak. The Transit Capacity and Quality of Service Manual uses segment based load factor as a measure of onboard loading comfort QoS. This paper provides additional insight into QoS by relating the two route based dimensions of occupancy load factor and passenger average travel time together in a two dimensional format, both from the passenger’s and operator’s perspectives. Future research will apply Value of Time to QoS measurement, reflecting perceived passenger comfort through crowding and average time spent onboard. This would also assist in transit service quality econometric modeling. The methodology can be readily applied in a practical setting where AFC data for fixed scheduled routes is available. The study outcomes also provide valuable research and development directions.

Association of hearing impairment and subsequent driving mobility in older adults

- Purpose of the study Hearing impairment (HI) is associated with driving safety (e.g., increased crashes and poor on-road driving performance). However, little is known about HI and driving mobility. This study examined the longitudinal association of audiometric hearing with older adults’ driving mobility over three years. - Design and Methods Secondary data analyses were conducted of 500 individuals (63-90 years) from the Staying Keen in Later Life (SKILL) study. Hearing (pure tone average of 0.5, 1, and 2 kHz) was assessed in the better hearing ear and categorized into normal hearing <25 dB HL; mild HI 26-40 dB HL; or moderate and greater HI>41 dB HL. The Useful Field of View Test (UFOV) was used to estimate the risk for adverse driving events. MANCOVA compared driving mobility between HI levels across time, adjusting for age, sex, race, hypertension, and stroke. Adjusting for these same covariates, Cox regression analyses examined incidence of driving cessation by HI across three years. - Results Individuals with moderate or greater HI performed poorly on the UFOV, indicating increased risk for adverse driving events (p<.001). No significant differences were found among older adults with varying levels of HI for driving mobility (ps>.05), including driving cessation rates (p=.38), across time. - Implications Although prior research indicates older adults with HI may be at higher risk for crashes, they may not modify driving over time. Further exploration of this issue is required to optimize efforts to improve driving safety and mobility among older adults.

Rainfall and streamflow response to El Niño Southern Oscillation: a case study in a semiarid catchment, Australia.

This paper aims to compare the shift in frequency distribution and skill of seasonal climate forecasting of both streamflow and rainfall in eastern Australia based on the Southern Oscillation Index (SOI) Phase system. Recent advances in seasonal forecasting of climate variables have highlighted opportunities for improving decision making in natural resources management. Forecasting of rainfall probabilities for different regions in Australia is available, but the use of similar forecasts for water resource supply has not been developed. The use of streamflow forecasts may provide better information for decision-making in irrigation supply and flow management for improved ecological outcomes. To examine the relative efficacy of seasonal forecasting of streamflow and rainfall, the shift in probability distributions and the forecast skill were evaluated using the Wilcoxon rank-sum test and the linear error in probability space (LEPS) skill score, respectively, at three river gauging stations in the Border Rivers Catchment of the Murray-Darling Basin in eastern Australia. A comparison of rainfall and streamflow distributions confirms higher statistical significance in the shift of streamflow distribution than that in rainfall distribution. Moreover, streamflow distribution showed greater skill of forecasting with 0-3 month lead time, compared to rainfall distribution.

Studies of working memory in interrupted human-computer interaction

Distraction in the workplace is increasingly more common in the information age. Several tasks and sources of information compete for a worker's limited cognitive capacities in human-computer interaction (HCI). In some situations even very brief interruptions can have detrimental effects on memory. Nevertheless, in other situations where persons are continuously interrupted, virtually no interruption costs emerge. This dissertation attempts to reveal the mental conditions and causalities differentiating the two outcomes. The explanation, building on the theory of long-term working memory (LTWM; Ericsson and Kintsch, 1995), focuses on the active, skillful aspects of human cognition that enable the storage of task information beyond the temporary and unstable storage provided by short-term working memory (STWM). Its key postulate is called a retrieval structure an abstract, hierarchical knowledge representation built into long-term memory that can be utilized to encode, update, and retrieve products of cognitive processes carried out during skilled task performance. If certain criteria of practice and task processing are met, LTWM allows for the storage of large representations for long time periods, yet these representations can be accessed with the accuracy, reliability, and speed typical of STWM. The main thesis of the dissertation is that the ability to endure interruptions depends on the efficiency in which LTWM can be recruited for maintaing information. An observational study and a field experiment provide ecological evidence for this thesis. Mobile users were found to be able to carry out heavy interleaving and sequencing of tasks while interacting, and they exhibited several intricate time-sharing strategies to orchestrate interruptions in a way sensitive to both external and internal demands. Interruptions are inevitable, because they arise as natural consequences of the top-down and bottom-up control of multitasking. In this process the function of LTWM is to keep some representations ready for reactivation and others in a more passive state to prevent interference. The psychological reality of the main thesis received confirmatory evidence in a series of laboratory experiments. They indicate that after encoding into LTWM, task representations are safeguarded from interruptions, regardless of their intensity, complexity, or pacing. However, when LTWM cannot be deployed, the problems posed by interference in long-term memory and the limited capacity of the STWM surface. A major contribution of the dissertation is the analysis of when users must resort to poorer maintenance strategies, like temporal cues and STWM-based rehearsal. First, one experiment showed that task orientations can be associated with radically different patterns of retrieval cue encodings. Thus the nature of the processing of the interface determines which features will be available as retrieval cues and which must be maintained by other means. In another study it was demonstrated that if the speed of encoding into LTWM, a skill-dependent parameter, is slower than the processing speed allowed for by the task, interruption costs emerge. Contrary to the predictions of competing theories, these costs turned out to involve intrusions in addition to omissions. Finally, it was learned that in rapid visually oriented interaction, perceptual-procedural expectations guide task resumption, and neither STWM nor LTWM are utilized due to the fact that access is too slow. These findings imply a change in thinking about the design of interfaces. Several novel principles of design are presented, basing on the idea of supporting the deployment of LTWM in the main task.

Motorcycle riders' self-reported aggression when riding compared with car driving

Aggressive driving has been shown to be related to increased crash risk for car driving. However, less is known about aggressive behaviour and motorcycle riding and whether there are differences in on-road aggression as a function of vehicle type. If such differences exist, these could relate to differences in perceptions of relative vulnerability associated with characteristics of the type of vehicle such as level of protection and performance. Specifically, the relative lack of protection offered by motorcycles may cause riders to feel more vulnerable and therefore to be less aggressive when they are riding compared to when they are driving. This study examined differences in self-reported aggression as a function of two vehicle types: passenger cars and motorcycles. Respondents (n = 247) were all motorcyclists who also drove a car. Results were that scores for the composite driving aggression scale were significantly higher than on the composite riding aggression scale. Regression analyses identified different patterns of predictors for driving aggression from those for riding aggression. Safety attitudes followed by thrill seeking tendencies were the strongest predictors for driving aggression, with more positive safety attitudes being protective while greater thrill seeking was associated with greater self-reported aggressive driving behaviour. For riding aggression, thrill seeking was the strongest predictor (positive relationship), followed by self-rated skill, such that higher self rated skill was protective against riding aggression. Participants who scored at the 85th percentile or above for the aggressive driving and aggressive riding indices had significantly higher scores on thrill seeking, greater intentions to engage in future risk taking, and lower safety attitude scores than other participants. In addition participants with the highest aggressive driving scores also had higher levels of self-reported past traffic offences than other participants. Collectively, these findings suggest that people are less likely to act aggressively when riding a motorcycle than when driving a car, and that those who are the most aggressive drivers are different from those who are the most aggressive riders. However, aggressive riders and drivers appear to present a risk to themselves and others on road. Importantly, the underlying influences for aggressive riding or driving that were identified in this study may be amenable to education and training interventions.

The regional differences between countries in traffic safety: A cross-cultural study and Turkish Case

Road traffic accidents are a large problem everywhere in the world. However, regional differences in traffic safety between countries are considerable. For example, traffic safety records are much worse in Southern Europe and the Middle East than in Northern and Western Europe. Despite the large regional differences in traffic safety, factors contributing to different accident risk figures in different countries and regions have remained largely unstudied. The general aim of this study was to investigate regional differences in traffic safety between Southern European/Middle Eastern (i.e., Greece, Iran, Turkey) and Northern/Western European (i.e., Finland, Great Britain, The Netherlands) countries and to identify factors related to these differences. We conducted seven sub-studies in which I applied a traffic culture framework, including a multi-level approach, to traffic safety. We used aggregated level data (national statistics), surveys among drivers, and data on traffic accidents and fatalities in the analyses. In the first study, we investigated the influence of macro level factors (i.e., economic, societal, and cultural) on traffic safety across countries. The results showed that a high GNP per capita and conservatism correlated with a low number of traffic fatalities, whereas a high degree of uncertainty avoidance, neuroticism, and egalitarianism correlated with a high number of traffic fatalities. In the second, third, and fourth studies, we examined whether the conceptualisation of road user characteristics (i.e., driver behaviour and performance) varied across traffic cultures and how these factors determined overall safety, and the differences between countries in traffic safety. The results showed that the factorial agreement for driver behaviour (i.e., aggressive driving) and performance (i.e., safety skills) was unsatisfactory in Greece, Iran, and Turkey, where the lack of social tolerance and interpersonal aggressive violations seem to be important characteristics of driving. In addition, we found that driver behaviour (i.e., aggressive violations and errors) mediated the relationship between culture/country and accidents. Besides, drivers from "dangerous" Southern European countries and Iran scored higher on aggressive violations and errors than did drivers from "safe" Northern European countries. However, "speeding" appeared to be a "pan-cultural" problem in traffic. Similarly, aggressive driving seems largely depend on road users' interactions and drivers' interpretation (i.e., cognitive biases) of the behaviour of others in every country involved in the study. Moreover, in all countries, a risky general driving style was mostly related to being young and male. The results of the fifth and sixth studies showed that among young Turkish drivers, gender stereotypes (i.e., masculinity and femininity) greatly influence driver behaviour and performance. Feminine drivers were safety-oriented whereas masculine drivers were skill-oriented and risky drivers. Since everyday driving tasks involve not only erroneous (i.e., risky or dangerous driving) or correct performance (i.e., normal habitual driving), but also "positive" driver behaviours, we developed a reliable scale for measuring "positive" driver behaviours among Turkish drivers in the seventh study. Consequently, I revised Reason's model [Reason, J. T., 1990. Human error. Cambridge University Press: New York] of aberrant driver behaviour to represent a general driving style, including all possible intentional behaviours in traffic while evaluating the differences between countries in traffic safety. The results emphasise the importance of economic, societal and cultural factors, general driving style and skills, which are related to exposure, cognitive biases as well as age, sex, and gender, in differences between countries in traffic safety.

From inferential statistics to climate knowledge

Climate variability and change are risk factors for climate sensitive activities such as agriculture. Managing these risks requires "climate knowledge", i.e. a sound understanding of causes and consequences of climate variability and knowledge of potential management options that are suitable in light of the climatic risks posed. Often such information about prognostic variables (e.g. yield, rainfall, run-off) is provided in probabilistic terms (e.g. via cumulative distribution functions, CDF), whereby the quantitative assessments of these alternative management options is based on such CDFs. Sound statistical approaches are needed in order to assess whether difference between such CDFs are intrinsic features of systems dynamics or chance events (i.e. quantifying evidences against an appropriate null hypothesis). Statistical procedures that rely on such a hypothesis testing framework are referred to as "inferential statistics" in contrast to descriptive statistics (e.g. mean, median, variance of population samples, skill scores). Here we report on the extension of some of the existing inferential techniques that provides more relevant and adequate information for decision making under uncertainty.

Probabilistic forecasts of the onset of the north Australian wet season

The amount and timing of early wet-season rainfall are important for the management of many agricultural industries in north Australia. With this in mind, a wet-season onset date is defined based on the accumulation of rainfall to a predefined threshold, starting from 1 September, for each square of a 1° gridded analysis of daily rainfall across the region. Consistent with earlier studies, the interannual variability of the onset dates is shown to be well related to the immediately preceding July-August Southern Oscillation index (SOI). Based on this relationship, a forecast method using logistic regression is developed to predict the probability that onset will occur later than the climatological mean date. This method is expanded to also predict the probabilities that onset will be later than any of a range of threshold dates around the climatological mean. When assessed using cross-validated hindcasts, the skill of the predictions exceeds that of climatological forecasts in the majority of locations in north Australia, especially in the Top End region, Cape York, and central Queensland. At times of strong anomalies in the July-August SOI, the forecasts are reliably emphatic. Furthermore, predictions using tropical Pacific sea surface temperatures (SSTs) as the predictor are also tested. While short-lead (July-August predictor) forecasts are more skillful using the SOI, long-lead (May-June predictor) forecasts are more skillful using Pacific SSTs, indicative of the longer-term memory present in the ocean.

Needs for applied climate education in agriculture

This paper reports on a purposive survey study which aimed to identify needs for the development, delivery and evaluation of applied climate education for targeted groups, to improve knowledge and skills to better manage under variable climatic conditions. The survey sample consisted of 80 producers and other industry stakeholders in Australia (including representatives from consulting, agricultural extension and agricultural education sectors), with a 58% response rate to the survey. The survey included an assessment of (i) knowledge levels of the Southern Oscillation Index and sea surface temperatures, and (ii) skill and ability in interpreting weather and climate parameters. Results showed that despite many of the respondents having more than 20 years experience in their industry, the only formal climate education or training undertaken by most was a 1-day workshop. Over 80% of the applied climate skills listed in the survey were regarded by respondents as essential or important, but only 42% of educators, 30% of consultants and 28% of producers rated themselves as competent in applying such skills. Essential skills were deemed as those that would enable respondents or their clients to be better prepared for the next extended wet or dry meteorological event, and improved capability in identifying and capitalising on key decision points from climate information and a seasonal climate outlook. The complex issue of forecast accuracy is a confounding obstacle for many in the application of climate information and forecasts in management. Addressing this problem by describing forecast 'limitations and skill' can help to overcome this problem. The survey also highlighted specific climatic tactical and strategic information collated from grazing, cropping and agribusiness enterprises, and showed the value of such information from a users perspective.

Constructing skilled images

When experts construct mental images, they do not rely only on perceptual features; they also access domain-specific knowledge and skills in long-term memory, which enables them to exceed the capacity limitations of the short-term working memory system. The central question of the present dissertation was whether the facilitating effect of long-term memory knowledge on working memory imagery tasks is primarily based on perceptual chunking or whether it relies on higher-level conceptual knowledge. Three domains of expertise were studied: chess, music, and taxi driving. The effects of skill level, stimulus surface features, and the stimulus structure on incremental construction of mental images were investigated. A method was developed to capture the chunking mechanisms that experts use in constructing images: chess pieces, street names, and visual notes were presented in a piecemeal fashion for later recall. Over 150 experts and non-experts participated in a total of 13 experiments, as reported in five publications. The results showed skill effects in all of the studied domains when experts performed memory and problem solving tasks that required mental imagery. Furthermore, only experts' construction of mental images benefited from meaningful stimuli. Manipulation of the stimulus surface features, such as replacing chess pieces with dots, did not significantly affect experts' performance in the imagery tasks. In contrast, the structure of the stimuli had a significant effect on experts' performance in every task domain. For example, taxi drivers recalled more street names from lists that formed a spatially continuous route than from alphabetically organised lists. The results suggest that the mechanisms of conceptual chunking rather than automatic perceptual pattern matching underlie expert performance, even though the tasks of the present studies required perception-like mental representations. The results show that experts are able to construct skilled images that surpass working memory capacity, and that their images are conceptually organised and interpreted rather than merely depictive.

Defining and predicting the 'break of the season' for north-east Queensland grazing areas

For pasture growth in the semi-arid tropics of north-east Australia, where up to 80% of annual rainfall occurs between December and March, the timing and distribution of rainfall events is often more important than the total amount. In particular, the timing of the 'green break of the season' (GBOS) at the end of the dry season, when new pasture growth becomes available as forage and a live-weight gain is measured in cattle, affects several important management decisions that prevent overgrazing and pasture degradation. Currently, beef producers in the region use a GBOS rule based on rainfall (e. g. 40mm of rain over three days by 1 December) to define the event and make their management decisions. A survey of 16 beef producers in north-east Queensland shows three quarters of respondents use a rainfall amount that occurs in only half or less than half of all years at their location. In addition, only half the producers expect the GBOS to occur within two weeks of the median date calculated by the CSIRO plant growth days model GRIM. This result suggests that in the producer rules, either the rainfall quantity or the period of time over which the rain is expected, is unrealistic. Despite only 37% of beef producers indicating that they use a southern oscillation index (SOI) forecast in their decisions, cross validated LEPS (linear error in probability space) analyses showed both the average 3 month July-September SOI and the 2 month August-September SOI have significant forecast skill in predicting the probability of both the amount of wet season rainfall and the timing of the GBOS. The communication and implementation of a rigorous and realistic definition of the GBOS, and the likely impacts of anthropogenic climate change on the region are discussed in the context of the sustainable management of northern Australian rangelands.

Liikkumisen ja kokemusten yhteys virtuaaliympäristöissä

The aim of this study was to build a model and analyze how users move in a virtual environment and to explore the experiential dimensions connected with different ways of moving. Due to the lack of previous research on this subject, this was an explorative study. This study also aimed to identify different ways how users move in virtual environments and the background variables connected to them. It was hypothesized that fluent movement in virtual environments is connected to high presence, skill and challenge assessments. Test participants (n = 68) were mostly highly educated young adults. A virtual environment was built using a CAVE -type virtual reality interface. The task was to search for objects that do not belong into a normal house. The participants movement in the virtual house was recorded on a computer. Movement was modelled using a cluster analysis of information entropy based movement measurements, acceleration, amount of stops and time spent being stationary. The experiential dimensions were measured using the EVEQ -questionnaire. We were able to identify four different ways of moving in virtual environments. In respect of background variables, the four groups differed only in the amount of weekly computer usage. However, fluent movement in virtual environments was connected to a high sense of presence. Furthermore, participants who moved fluently in the environment assessed their skills as being high and regarded the use of virtual environment as challenging. The results indicate that different ways of moving affects how people experience virtual environments. Consequently the participants assessment of their skills and level of challenge have an impact on the affective evaluation of the situation at hand. Entropy measures have not been previously applied when studying movement, and in addition the role of movement on the experiential dimensions of virtual environments is an unexplored subject. The movement analysis method introduced here is applicable to other research problems. Finally, this study expands on our knowledge of the special characteristics connected with the experiential dimensions of virtual environments.