Evaluating artistic work : balancing competing perspectives

The artist is frequently uncertain, when he or she begins to create a work, how the completed work will look or sound. However, the corporate business model, which is premised on a rational and instrumental worldview, suggests that in a market environment, art should be evaluated objectively, based on clearly stated and measurable objectives - often prior to that work being commenced. This paper explores the difficulties that art has in fitting into a corporatist worldview. First, the paper examines the historical materialization of the corporate model, and how it has infiltrated non-profit arts. Second, the paper investigates the likely reasons as to why instrumental rationality and managerialism have been embraced so enthusiastically by bureaucrats, arts marketers and funders. And third, the paper suggests a research approach by which artists, managers and audiences can evaluate art within a framework that is sympathetic to the art and the artist.

New evaluation methodologies for public art programs in cultural instiutions

I put my abstract in for this conference back in March, based on some evaluation work I had been doing in 2010 with my colleague Professor Greg Hearn for the 3C Regional Writing NeoGreography Project. I had been swapping notes with a colleague from the Smithsonian’s Centre for Folklife and Cultural Heritage about their evaluation work, and stuck inside during the rains of January, I decided to apply for a Qld Smithsonian fellowship based on the quandary of evaluation-particular in public histories (oral histories) and digital storytelling. In July I was awarded the fellowship, so I have tweaked my presentation to talk about what we hope to do with this collaboration, to propel the importance placed on evaluation in public arts programs in Qld and beyond.

The art of interpretation: Tracing logics of evaluation in Shaun Tan’s The Lost Thing (2000) and Andrew Ruhemann and Shaun Tan’s The Lost Thing (2010)

Celebration (and the celebritisation) of the Australian-ness of children’s authors who enjoy critical or commercial international success, and especially of those who win international prizes speaks to a desire to partake in both national and international cultural spheres. Prizing is often presumed to both guarantee and emerge from a creator's reputation at home and abroad. Australian artist and writer Shaun Tan has received a wide array of cultural and literary prizes, ranging from Australian book awards, to an Academy Award, to the Astrid Lindgren Memorial Prize. This paper considers logics of evaluation and interpretation as they can be traced in the intratextual, intertextual, and extratextual codes of Shaun Tan’s picture book, The Lost Thing (2000), the animated film adaptation of The Lost Thing (2010). It further considers the ways in which the desire for a global audience may necessitate an erasure of the national culture which is traded on in a global market.

Automatic Tonic Identification in Indian Art Music: Approaches and Evaluation

The tonic is a fundamental concept in Indian art music. It is the base pitch, which an artist chooses in order to construct the melodies during a rg(a) rendition, and all accompanying instruments are tuned using the tonic pitch. Consequently, tonic identification is a fundamental task for most computational analyses of Indian art music, such as intonation analysis, melodic motif analysis and rg recognition. In this paper we review existing approaches for tonic identification in Indian art music and evaluate them on six diverse datasets for a thorough comparison and analysis. We study the performance of each method in different contexts such as the presence/absence of additional metadata, the quality of audio data, the duration of audio data, music tradition (Hindustani/Carnatic) and the gender of the singer (male/female). We show that the approaches that combine multi-pitch analysis with machine learning provide the best performance in most cases (90% identification accuracy on average), and are robust across the aforementioned contexts compared to the approaches based on expert knowledge. In addition, we also show that the performance of the latter can be improved when additional metadata is available to further constrain the problem. Finally, we present a detailed error analysis of each method, providing further insights into the advantages and limitations of the methods.

Evaluation of Earth system models using modern and palaeo-observations: the state-of-the-art

Earth system models are increasing in complexity and incorporating more processes than their predecessors, making them important tools for studying the global carbon cycle. However, their coupled behaviour has only recently been examined in any detail, and has yielded a very wide range of outcomes, with coupled climate-carbon cycle models that represent land-use change simulating total land carbon stores by 2100 that vary by as much as 600 Pg C given the same emissions scenario. This large uncertainty is associated with differences in how key processes are simulated in different models, and illustrates the necessity of determining which models are most realistic using rigorous model evaluation methodologies. Here we assess the state-of-the-art with respect to evaluation of Earth system models, with a particular emphasis on the simulation of the carbon cycle and associated biospheric processes. We examine some of the new advances and remaining uncertainties relating to (i) modern and palaeo data and (ii) metrics for evaluation, and discuss a range of strategies, such as the inclusion of pre-calibration, combined process- and system-level evaluation, and the use of emergent constraints, that can contribute towards the development of more robust evaluation schemes. An increasingly data-rich environment offers more opportunities for model evaluation, but it is also a challenge, as more knowledge about data uncertainties is required in order to determine robust evaluation methodologies that move the field of ESM evaluation from "beauty contest" toward the development of useful constraints on model behaviour.

Evaluation of biospheric components in Earth system models using modern and palaeo-observations: the state-of-the-art

Earth system models (ESMs) are increasing in complexity by incorporating more processes than their predecessors, making them potentially important tools for studying the evolution of climate and associated biogeochemical cycles. However, their coupled behaviour has only recently been examined in any detail, and has yielded a very wide range of outcomes. For example, coupled climate–carbon cycle models that represent land-use change simulate total land carbon stores at 2100 that vary by as much as 600 Pg C, given the same emissions scenario. This large uncertainty is associated with differences in how key processes are simulated in different models, and illustrates the necessity of determining which models are most realistic using rigorous methods of model evaluation. Here we assess the state-of-the-art in evaluation of ESMs, with a particular emphasis on the simulation of the carbon cycle and associated biospheric processes. We examine some of the new advances and remaining uncertainties relating to (i) modern and palaeodata and (ii) metrics for evaluation. We note that the practice of averaging results from many models is unreliable and no substitute for proper evaluation of individual models. We discuss a range of strategies, such as the inclusion of pre-calibration, combined process- and system-level evaluation, and the use of emergent constraints, that can contribute to the development of more robust evaluation schemes. An increasingly data-rich environment offers more opportunities for model evaluation, but also presents a challenge. Improved knowledge of data uncertainties is still necessary to move the field of ESM evaluation away from a "beauty contest" towards the development of useful constraints on model outcomes.

An evaluation of decadal probability forecasts from state-of-the-art climate models

While state-of-the-art models of Earth's climate system have improved tremendously over the last 20 years, nontrivial structural flaws still hinder their ability to forecast the decadal dynamics of the Earth system realistically. Contrasting the skill of these models not only with each other but also with empirical models can reveal the space and time scales on which simulation models exploit their physical basis effectively and quantify their ability to add information to operational forecasts. The skill of decadal probabilistic hindcasts for annual global-mean and regional-mean temperatures from the EU Ensemble-Based Predictions of Climate Changes and Their Impacts (ENSEMBLES) project is contrasted with several empirical models. Both the ENSEMBLES models and a “dynamic climatology” empirical model show probabilistic skill above that of a static climatology for global-mean temperature. The dynamic climatology model, however, often outperforms the ENSEMBLES models. The fact that empirical models display skill similar to that of today's state-of-the-art simulation models suggests that empirical forecasts can improve decadal forecasts for climate services, just as in weather, medium-range, and seasonal forecasting. It is suggested that the direct comparison of simulation models with empirical models becomes a regular component of large model forecast evaluations. Doing so would clarify the extent to which state-of-the-art simulation models provide information beyond that available from simpler empirical models and clarify current limitations in using simulation forecasting for decision support. Ultimately, the skill of simulation models based on physical principles is expected to surpass that of empirical models in a changing climate; their direct comparison provides information on progress toward that goal, which is not available in model–model intercomparisons.

Clinical evaluation of multisurface ART restorations

O objetivo deste estudo foi avaliar o comportamento de dois cimentos de ionômero de vidro: um de alta viscosidade (Ketac Molar - ESPE) e um modificado por resina (Fuji VIII – GC) em restaurações envolvendo duas ou mais superfícies dentárias, confeccionadas pela técnica do Tratamento Restaurador Atraumático. Sessenta restaurações (30 com cada material) foram inseridas em estudantes (9-16 anos) por dois operadores. Após 6 meses, dois examinadores independentes avaliaram as restaurações de acordo com os critérios utilizados em trabalhos prévios semelhantes. Os dados foram analisados pelos testes de McNemar e Fischer. A porcentagem de sucesso do tratamento foi de 98,3%. Uma restauração (Ketac Molar) foi substituída por outro material e classificada como falha. As porcentagens de sucesso das restaurações foram de 100% e 96,6% para o Fuji VIII e Ketac Molar, respectivamente. Não houve diferença estatisticamente significante no sucesso das restaurações entre o baseline e 6 meses (p>0,05). Da mesma forma, não houve diferença estatística entre os materiais, tipos de cavidade ou entre operadores.(p>0,05). A técnica ART foi altamente apropriada e efetiva em restaurações envolvendo duas ou mais superfícies, após 6 meses. Os resultados mostraram um comportamento promissor com ambos os materiais.

Clinical evaluation of multiple-surface ART restorations: three-year follow-up

Fundação de Amparo à Pesquisa do Estado de São Paulo (FAPESP)

Evaluation of the Design of Komendant's Cycloidal, Thin Shell Roofs At the Kimbell Art Museum Using Finite Element Analysis

The analysis of Komendant's design of the Kimbell Art Museum was carried out in order to determine the effectiveness of the ring beams, edge beams and prestressing in the shells of the roof system. Finite element analysis was not available to Komendant or other engineers of the time to aid them in the design and analysis. Thus, the use of this tool helped to form a new perspective on the Kimbell Art Museum and analyze the engineer's work. In order to carry out the finite element analysis of Kimbell Art Museum, ADINA finite element analysis software was utilized. Eight finite element models (FEM-1 through FEM-8) of increasing complexity were created. The results of the most realistic model, FEM-8, which included ring beams, edge beams and prestressing, were compared to Komendant's calculations. The maximum deflection at the crown of the mid-span surface of -0.1739 in. in FEM-8 was found to be larger than Komendant's deflection in the design documents before the loss in prestressing force (-0.152 in.) but smaller than his prediction after the loss in prestressing force (-0.3814 in.). Komendant predicted a larger longitudinal stress of -903 psi at the crown (vs. -797 psi in FEM-8) and 37 psi at the edge (vs. -347 psi in FEM-8). Considering the strength of concrete of 5000 psi, the difference in results is not significant. From the analysis it was determined that both FEM-5, which included prestressing and fixed rings, and FEM-8 can be successfully and effectively implemented in practice. Prestressing was used in both models and thus served as the main contribution to efficiency. FEM-5 showed that ring and edge beams can be avoided, however an architect might find them more aesthetically appropriate than rigid walls.

Evaluation of Three State-of-the-Art Classifiers for Recognition of Activities of Daily Living from Smart Home Ambient Data

Smart homes for the aging population have recently started attracting the attention of the research community. The "health state" of smart homes is comprised of many different levels; starting with the physical health of citizens, it also includes longer-term health norms and outcomes, as well as the arena of positive behavior changes. One of the problems of interest is to monitor the activities of daily living (ADL) of the elderly, aiming at their protection and well-being. For this purpose, we installed passive infrared (PIR) sensors to detect motion in a specific area inside a smart apartment and used them to collect a set of ADL. In a novel approach, we describe a technology that allows the ground truth collected in one smart home to train activity recognition systems for other smart homes. We asked the users to label all instances of all ADL only once and subsequently applied data mining techniques to cluster in-home sensor firings. Each cluster would therefore represent the instances of the same activity. Once the clusters were associated to their corresponding activities, our system was able to recognize future activities. To improve the activity recognition accuracy, our system preprocessed raw sensor data by identifying overlapping activities. To evaluate the recognition performance from a 200-day dataset, we implemented three different active learning classification algorithms and compared their performance: naive Bayesian (NB), support vector machine (SVM) and random forest (RF). Based on our results, the RF classifier recognized activities with an average specificity of 96.53%, a sensitivity of 68.49%, a precision of 74.41% and an F-measure of 71.33%, outperforming both the NB and SVM classifiers. Further clustering markedly improved the results of the RF classifier. An activity recognition system based on PIR sensors in conjunction with a clustering classification approach was able to detect ADL from datasets collected from different homes. Thus, our PIR-based smart home technology could improve care and provide valuable information to better understand the functioning of our societies, as well as to inform both individual and collective action in a smart city scenario.