Statistical hypothesis testing and common misinterpretations: should we abandon p-value in forensic science applications?

Many people regard the concept of hypothesis testing as fundamental to inferential statistics. Various schools of thought, in particular frequentist and Bayesian, have promoted radically different solutions for taking a decision about the plausibility of competing hypotheses. Comprehensive philosophical comparisons about their advantages and drawbacks are widely available and continue to span over large debates in the literature. More recently, controversial discussion was initiated by an editorial decision of a scientific journal [1] to refuse any paper submitted for publication containing null hypothesis testing procedures. Since the large majority of papers published in forensic journals propose the evaluation of statistical evidence based on the so called p-values, it is of interest to expose the discussion of this journal's decision within the forensic science community. This paper aims to provide forensic science researchers with a primer on the main concepts and their implications for making informed methodological choices.

Hypothesis testing and earthquake prediction.

Requirements for testing include advance specification of the conditional rate density (probability per unit time, area, and magnitude) or, alternatively, probabilities for specified intervals of time, space, and magnitude. Here I consider testing fully specified hypotheses, with no parameter adjustments or arbitrary decisions allowed during the test period. Because it may take decades to validate prediction methods, it is worthwhile to formulate testable hypotheses carefully in advance. Earthquake prediction generally implies that the probability will be temporarily higher than normal. Such a statement requires knowledge of "normal behavior"--that is, it requires a null hypothesis. Hypotheses can be tested in three ways: (i) by comparing the number of actual earth-quakes to the number predicted, (ii) by comparing the likelihood score of actual earthquakes to the predicted distribution, and (iii) by comparing the likelihood ratio to that of a null hypothesis. The first two tests are purely self-consistency tests, while the third is a direct comparison of two hypotheses. Predictions made without a statement of probability are very difficult to test, and any test must be based on the ratio of earthquakes in and out of the forecast regions.

Conducting statistical tests of hypotheses : five common misconceptions found in transportation research

Most statistical methods use hypothesis testing. Analysis of variance, regression, discrete choice models, contingency tables, and other analysis methods commonly used in transportation research share hypothesis testing as the means of making inferences about the population of interest. Despite the fact that hypothesis testing has been a cornerstone of empirical research for many years, various aspects of hypothesis tests commonly are incorrectly applied, misinterpreted, and ignored—by novices and expert researchers alike. On initial glance, hypothesis testing appears straightforward: develop the null and alternative hypotheses, compute the test statistic to compare to a standard distribution, estimate the probability of rejecting the null hypothesis, and then make claims about the importance of the finding. This is an oversimplification of the process of hypothesis testing. Hypothesis testing as applied in empirical research is examined here. The reader is assumed to have a basic knowledge of the role of hypothesis testing in various statistical methods. Through the use of an example, the mechanics of hypothesis testing is first reviewed. Then, five precautions surrounding the use and interpretation of hypothesis tests are developed; examples of each are provided to demonstrate how errors are made, and solutions are identified so similar errors can be avoided. Remedies are provided for common errors, and conclusions are drawn on how to use the results of this paper to improve the conduct of empirical research in transportation.

An analysis of holding cost impact on housing affordability in relation to midsized Greenfield residential property developments in South East Queensland

The position of housing demand and supply is not consistent. The Australian situation counters the experience demonstrated in many other parts of the world in the aftermath of the Global Financial Crisis, with residential housing prices proving particularly resilient. A seemingly inexorable housing demand remains a critical issue affecting the socio-economic landscape. Underpinned by high levels of population growth fuelled by immigration, and further buoyed by sustained historically low interest rates, increasing income levels, and increased government assistance for first home buyers, this strong housing demand level ensures problems related to housing affordability continue almost unabated. A significant, but less visible factor impacting housing affordability relates to holding costs. Although only one contributor in the housing affordability matrix, the nature and extent of holding cost impact requires elucidation: for example, the computation and methodology behind the calculation of holding costs varies widely - and in some instances completely ignored. In addition, ambiguity exists in terms of the inclusion of various elements that comprise holding costs, thereby affecting the assessment of their relative contribution. Such anomalies may be explained by considering that assessment is conducted over time in an ever-changing environment. A strong relationship with opportunity cost - in turn dependant inter alia upon prevailing inflation and / or interest rates - adds further complexity. By extending research in the general area of housing affordability, this thesis seeks to provide a detailed investigation of those elements related to holding costs specifically in the context of midsized (i.e. between 15-200 lots) greenfield residential property developments in South East Queensland. With the dimensions of holding costs and their influence over housing affordability determined, the null hypothesis H0 that holding costs are not passed on can be addressed. Arriving at these conclusions involves the development of robust economic and econometric models which seek to clarify the componentry impacts of holding cost elements. An explanatory sequential design research methodology has been adopted, whereby the compilation and analysis of quantitative data and the development of an economic model is informed by the subsequent collection and analysis of primarily qualitative data derived from surveying development related organisations. Ultimately, there are significant policy implications in relation to the framework used in Australian jurisdictions that promote, retain, or otherwise maximise, the opportunities for affordable housing.

Measuring voluntary dietary change in response to exercise : a focus on dietary fat

Traditional treatments for weight management have focussed on prescribed dietary restriction or regular exercise, or a combination of both. However recidivism for such prescribed treatments remains high, particularly among the overweight and obese. The aim of this thesis was to investigate voluntary dietary changes in the presence of prescribed mixed-mode exercise, conducted over 16 weeks. With the implementation of a single lifestyle change (exercise) it was postulated that the onerous burden of concomitant dietary and exercise compliance would be reduced, leading to voluntary lifestyle changes in such areas as diet. In addition, the failure of exercise as a single weight loss treatment has been reported to be due to compensatory energy intakes, although much of the evidence is from acute exercise studies, necessitating investigation of compensatory intakes during a long-term exercise intervention. Following 16 weeks of moderate intensity exercise, 30 overweight and obese (BMI≥25.00 kg.m-2) men and women showed small but statistically significant decreases in mean dietary fat intakes, without compensatory increases in other macronutrient or total energy intakes. Indeed total energy intakes were significantly lower for men and women following the exercise intervention, due to the decreases in dietary fat intakes. There was a risk that acceptance of the statistical validity of the small changes to dietary fat intakes may have constituted a Type 1 error, with false rejection of the Null hypothesis. Oro-sensory perceptions to changes in fat loads were therefore investigated to determine whether the measured dietary fat changes were detectable by the human palate. The ability to detect small changes in dietary fat provides sensory feedback for self-initiated dietary changes, but lean and overweight participants were unable to distinguish changes to fat loads of similar magnitudes to that measured in the exercise intervention study. Accuracy of the dietary measurement instrument was improved with the effects of random error (day-to-day variability) minimised with the use of a statistically validated 8-day, multiple-pass, 24 hour dietary recall instrument. However systematic error (underreporting) may have masked the magnitude of dietary change, particularly the reduction in dietary fat intakes. A purported biomarker (plasma Apolipoprotein A-IV) (apoA-IV) was subsequently investigated, to monitor systematic error in self-reported dietary intakes. Changes in plasma apoA-IV concentrations were directly correlated with increased and decreased changes to dietary fat intakes, suggesting that this objective marker may be a useful tool to improve the accuracy of dietary measurement in overweight and obese populations, who are susceptible to dietary underreporting.

A smooth transition logit model of the effects of deregulation in the electricity market

This paper introduces the smooth transition logit (STL) model that is designed to detect and model situations in which there is structural change in the behaviour underlying the latent index from which the binary dependent variable is constructed. The maximum likelihood estimators of the parameters of the model are derived along with their asymptotic properties, together with a Lagrange multiplier test of the null hypothesis of linearity in the underlying latent index. The development of the STL model is motivated by the desire to assess the impact of deregulation in the Queensland electricity market and ascertain whether increased competition has resulted in significant changes in the behaviour of the spot price of electricity, specifically with respect to the occurrence of periodic abnormally high prices. The model allows the timing of any change to be endogenously determined and also market participants' behaviour to change gradually over time. The main results provide clear evidence in support of a structural change in the nature of price events, and the endogenously determined timing of the change is consistent with the process of deregulation in Queensland.

From inferential statistics to climate knowledge

Climate variability and change are risk factors for climate sensitive activities such as agriculture. Managing these risks requires "climate knowledge", i.e. a sound understanding of causes and consequences of climate variability and knowledge of potential management options that are suitable in light of the climatic risks posed. Often such information about prognostic variables (e.g. yield, rainfall, run-off) is provided in probabilistic terms (e.g. via cumulative distribution functions, CDF), whereby the quantitative assessments of these alternative management options is based on such CDFs. Sound statistical approaches are needed in order to assess whether difference between such CDFs are intrinsic features of systems dynamics or chance events (i.e. quantifying evidences against an appropriate null hypothesis). Statistical procedures that rely on such a hypothesis testing framework are referred to as "inferential statistics" in contrast to descriptive statistics (e.g. mean, median, variance of population samples, skill scores). Here we report on the extension of some of the existing inferential techniques that provides more relevant and adequate information for decision making under uncertainty.

Patterns in the distribution and abundance of female eastern king prawns, Melicertus plebejus (Hess, 1865), capable of spawning and reproductive potential in waters off

Linear mixed models were used to test the null hypothesis that there were no differences between seasons and locations in the reproductive potential of female eastern king prawns, Melicertus plebejus along the east coast of Australia. Three samples were collected in each season between autumn 1991 and winter 1992 (inclusive). Females capable of spawning were found at all locations but proportions were greater in lower than higher latitudes. Females capable of spawning were not found at the southern (highest latitude) most location in all seasons. There was a significant interaction in reproductive potential between seasons and locations suggesting that patterns among seasons differed between locations and vice versa. Reproductive potential was greatest amongst the northern (lower latitudes) most locations and was greatest in autumn at these locations. Seasonal patterns were less pronounced further south (higher latitudes). The length composition of females in catches differed between locations with more larger prawns found in samples from northern locations. The challenge that remains is to quantify the oceanic sources of larvae that contribute to recruitment in each nursery area and the estuarine sources of juveniles that contribute adults back to the effective spawning stock. Maintaining the effective spawning stock and important nursery areas are crucial to the sustainability of this resource.

Scalloped Hammerhead microsatellite genotypes (eight loci) representing five regions on Australia's east coast and one location in central Indonesia.

Biodiversity of sharks in the tropical Indo-Pacific is high, but species-specific information to assist sustainable resource exploitation is scarce. The null hypothesis of population genetic homogeneity was tested for scalloped hammerhead shark (Sphyrna lewini, n=244) and the milkshark (Rhizoprionodon acutus, n=209) from northern and eastern Australia, using nuclear (S. lewini, eight microsatellite loci; R. acutus, six loci) and mitochondrial gene markers (873 base pairs of NADH dehydrogenase subunit 4). We were unable to reject genetic homogeneity for S. lewini, which was as expected based on previous studies of this species. Less expected were similar results for R. acutus, which is more benthic and less vagile than S. lewini. These features are probably driving the genetic break found between Australian and central Indonesian R. acutus (F-statistics; mtDNA, 0.751 to 0.903; microsatellite loci, 0.038 to 0.047). Our results support the spatially-homogeneous management plan for shark species in Queensland, but caution is advised for species yet to be studied.

Milkshark (Rhizoprionodon acutus) microsatellite genotype data (six loci) from samples collected from five locations in eastern Australia and central Indonesia.

Biodiversity of sharks in the tropical Indo-Pacific is high, but species-specific information to assist sustainable resource exploitation is scarce. The null hypothesis of population genetic homogeneity was tested for scalloped hammerhead shark (Sphyrna lewini, n=244) and the milkshark (Rhizoprionodon acutus, n=209) from northern and eastern Australia, using nuclear (S. lewini, eight microsatellite loci; R. acutus, six loci) and mitochondrial gene markers (873 base pairs of NADH dehydrogenase subunit 4). We were unable to reject genetic homogeneity for S. lewini, which was as expected based on previous studies of this species. Less expected were similar results for R. acutus, which is more benthic and less vagile than S. lewini. These features are probably driving the genetic break found between Australian and central Indonesian R. acutus (F-statistics; mtDNA, 0.751 to 0.903; microsatellite loci, 0.038 to 0.047). Our results support the spatially-homogeneous management plan for shark species in Queensland, but caution is advised for species yet to be studied.

Negligible evidence for regional genetic population structure for two shark species Rhizoprionodon acutus (Rüppell, 1837) and Sphyrna lewini (Griffith & Smith, 1834) with contrasting biology

Biodiversity of sharks in the tropical Indo-Pacific is high, but species-specific information to assist sustainable resource exploitation is scarce. The null hypothesis of population genetic homogeneity was tested for scalloped hammerhead shark (Sphyrna lewini, n = 237) and the milk shark (Rhizoprionodon acutus, n = 207) from northern and eastern Australia, using nuclear (S. lewini, eight microsatellite loci; R. acutus, six loci) and mitochondrial gene markers (873 base pairs of NADH dehydrogenase subunit 4). We were unable to reject genetic homogeneity for S. lewini, which was as expected based on previous studies of this species. Less expected were similar results for R. acutus, which is more benthic and less vagile than S. lewini. These features are probably driving the genetic break found between Australian and central Indonesian R. acutus (F-statistics; mtDNA, 0.751–0.903, respectively; microsatellite loci, 0.038–0.047 respectively). Our results support the spatially homogeneous monitoring and management plan for shark species in Queensland, Australia.

Determination of management units for grey mackerel fisheries in northern Australia.

The requirement for Queensland, Northern Territory and Western Australian jurisdictions to ensure sustainable harvest of fish resources and their optimal use relies on robust information on the resource status. For grey mackerel (Scomberomorus semifasciatus) fisheries, each of these jurisdictions has their own management regime in their corresponding waters. The lack of information on stock structure of grey mackerel, however, means that the appropriate spatial scale of management is not known. As well, fishers require assurance of future sustainability to encourage investment and long-term involvement in a fishery that supplies lucrative overseas markets. These management and fisher-unfriendly circumstances must be viewed in the context of recent 3-fold increases in catches of grey mackerel along the Queensland east coast, combined with significant and increasing catches in other parts of the species' northern Australian range. Establishing the stock structure of grey mackerel would also immensely improve the relevance of resource assessments for fishery management of grey mackerel across northern Australia. This highlighted the urgent need for stock structure information for this species. The impetus for this project came from the strategic recommendations of the FRDC review by Ward and Rogers (2003), "Northern mackerel (Scombridae: Scomberomorus): current and future research needs" (Project No. 2002/096), which promoted the urgency for information on the stock structure of grey mackerel. In following these recommendations this project adopted a multi-technique and phased sampling approach as carried out by Buckworth et al (2007), who examined the stock structure of Spanish mackerel, Scomberomorus commerson, across northern Australia. The project objectives were to determine the stock structure of grey mackerel across their northern Australian range, and use this information to define management units and their appropriate spatial scales. We used multiple techniques concurrently to determine the stock structure of grey mackerel. These techniques were: genetic analyses (mitochondrial DNA and microsatellite DNA), otolith (ear bones) isotope ratios, parasite abundances, and growth parameters. The advantage of using this type of multi-technique approach was that each of the different methods is informative about the fish’s life history at different spatial and temporal scales. Genetics can inform about the evolutionary patterns as well as rates of mixing of fish from adjacent areas, while parasites and otolith microchemistry are directly influenced by the environment and so will inform about the patterns of movement during the fishes lifetime. Growth patterns are influenced by both genetic and environmental factors. Due to these differences the use of these techniques concurrently increases the likelihood of detecting different stocks where they exist. We adopted a phased sampling approach whereby sampling was carried out at broad spatial scales in the first year: east coast, eastern Gulf of Carpentaria (GoC), western GoC, and the NW Northern Territory (NW NT). By comparing the fish samples from each of these locations, and using each of the techniques, we tested the null hypothesis that grey mackerel were comprised of a single homogeneous population across northern Australia. Having rejected the null hypothesis we re-sampled the 1st year locations to test for temporal stability in stock structure, and to assess stock structure at finer spatial scales. This included increased spatial coverage on the east coast, the GoC, and WA. From genetic approaches we determined that there at least four genetic stocks of grey mackerel across northern Australia: WA, NW NT (Timor/Arafura), the GoC and the east Grey mackerel management units in northern Australia ix coast. All markers revealed concordant patterns showing WA and NW NT to be clearly divergent stocks. The mtDNA D-loop fragment appeared to have more power to resolve stock boundaries because it was able to show that the GoC and east coast QLD stocks were genetically differentiated. Patterns of stock structure on a finer scale, or where stock boundaries are located, were less clear. From otolith stable isotope analyses four major groups of S. semifasciatus were identified: WA, NT/GoC, northern east coast and central east coast. Differences in the isotopic composition of whole otoliths indicate that these groups must have spent their life history in different locations. The magnitude of the difference between the groups suggests a prolonged separation period at least equal to the fish’s life span. The parasite abundance analyses, although did not include samples from WA, suggest the existence of at least four stocks of grey mackerel in northern Australia: NW NT, the GoC, northern east coast and central east coast. Grey mackerel parasite fauna on the east coast suggests a separation somewhere between Townsville and Mackay. The NW NT region also appears to comprise a separate stock while within the GoC there exists a high degree of variability in parasite faunas among the regions sampled. This may be due to 1. natural variation within the GoC and there is one grey mackerel stock, or 2. the existence of multiple localised adult sub-stocks (metapopulations) within the GoC. Growth parameter comparisons were only possible from four major locations and identified the NW NT, the GoC, and the east coast as having different population growth characteristics. Through the use of multiple techniques, and by integrating the results from each, we were able to determine that there exist at least five stocks of grey mackerel across northern Australia, with some likelihood of additional stock structuring within the GoC. The major management units determined from this study therefore were Western Australia, NW Northern Territory (Timor/Arafura), the Gulf of Carpentaria, northern east Queensland coast and central east Queensland coast. The management implications of these results indicate the possible need for management of grey mackerel fisheries in Australia to be carried out on regional scales finer than are currently in place. In some regions the spatial scales of management might continue as is currently (e.g. WA), while in other regions, such as the GoC and the east coast, managers should at least monitor fisheries on a more local scale dictated by fishing effort and assess accordingly. Stock assessments should also consider the stock divisions identified, particularly on the east coast and for the GoC, and use life history parameters particular to each stock. We also emphasise that where we have not identified different stocks does not preclude the possibility of the occurrence of further stock division. Further, this study did not, nor did it set out to, assess the status of each of the stocks identified. This we identify as a high priority action for research and development of grey mackerel fisheries, as well as a management strategy evaluation that incorporates the conclusions of this work. Until such time that these priorities are addressed, management of grey mackerel fisheries should be cognisant of these uncertainties, particularly for the GoC and the Queensland east coast.

Statistical significance of sequential firing patterns in multi-neuronal spike trains

Sequential firings with fixed time delays are frequently observed in simultaneous recordings from multiple neurons. Such temporal patterns are potentially indicative of underlying microcircuits and it is important to know when a repeatedly occurring pattern is statistically significant. These sequences are typically identified through correlation counts. In this paper we present a method for assessing the significance of such correlations. We specify the null hypothesis in terms of a bound on the conditional probabilities that characterize the influence of one neuron on another. This method of testing significance is more general than the currently available methods since under our null hypothesis we do not assume that the spiking processes of different neurons are independent. The structure of our null hypothesis also allows us to rank order the detected patterns. We demonstrate our method on simulated spike trains.

Conditional Probability-Based Significance Tests for Sequential Patterns in Multineuronal Spike Trains

We consider the problem of detecting statistically significant sequential patterns in multineuronal spike trains. These patterns are characterized by ordered sequences of spikes from different neurons with specific delays between spikes. We have previously proposed a data-mining scheme to efficiently discover such patterns, which occur often enough in the data. Here we propose a method to determine the statistical significance of such repeating patterns. The novelty of our approach is that we use a compound null hypothesis that not only includes models of independent neurons but also models where neurons have weak dependencies. The strength of interaction among the neurons is represented in terms of certain pair-wise conditional probabilities. We specify our null hypothesis by putting an upper bound on all such conditional probabilities. We construct a probabilistic model that captures the counting process and use this to derive a test of significance for rejecting such a compound null hypothesis. The structure of our null hypothesis also allows us to rank-order different significant patterns. We illustrate the effectiveness of our approach using spike trains generated with a simulator.

Testing against changing correlation

A test for time-varying correlation is developed within the framework of a dynamic conditional score (DCS) model for both Gaussian and Student t-distributions. The test may be interpreted as a Lagrange multiplier test and modified to allow for the estimation of models for time-varying volatility in the individual series. Unlike standard moment-based tests, the score-based test statistic includes information on the level of correlation under the null hypothesis and local power arguments indicate the benefits of doing so. A simulation study shows that the performance of the score-based test is strong relative to existing tests across a range of data generating processes. An application to the Hong Kong and South Korean equity markets shows that the new test reveals changes in correlation that are not detected by the standard moment-based test.