Eliciting and Aggregating Forecasts When Information is Shared

Using the wisdom of crowds---combining many individual forecasts to obtain an aggregate estimate---can be an effective technique for improving forecast accuracy. When individual forecasts are drawn from independent and identical information sources, a simple average provides the optimal crowd forecast. However, correlated forecast errors greatly limit the ability of the wisdom of crowds to recover the truth. In practice, this dependence often emerges because information is shared: forecasters may to a large extent draw on the same data when formulating their responses.

To address this problem, I propose an elicitation procedure in which each respondent is asked to provide both their own best forecast and a guess of the average forecast that will be given by all other respondents. I study optimal responses in a stylized information setting and develop an aggregation method, called pivoting, which separates individual forecasts into shared and private information and then recombines these results in the optimal manner. I develop a tailored pivoting procedure for each of three information models, and introduce a simple and robust variant that outperforms the simple average across a variety of settings.

In three experiments, I investigate the method and the accuracy of the crowd forecasts. In the first study, I vary the shared and private information in a controlled environment, while the latter two studies examine forecasts in real-world contexts. Overall, the data suggest that a simple minimal pivoting procedure provides an effective aggregation technique that can significantly outperform the crowd average.

Correcting the errors: Volatility forecast evaluation using high-frequency data and realized volatilities

We develop general model-free adjustment procedures for the calculation of unbiased volatility loss functions based on practically feasible realized volatility benchmarks. The procedures, which exploit recent nonparametric asymptotic distributional results, are both easy-to-implement and highly accurate in empirically realistic situations. We also illustrate that properly accounting for the measurement errors in the volatility forecast evaluations reported in the existing literature can result in markedly higher estimates for the true degree of return volatility predictability.

Can exchange rates forecast commodity prices?

We show that "commodity currency" exchange rates have surprisingly robust power in predicting global commodity prices, both in-sample and out-of-sample, and against a variety of alternative benchmarks. This result is of particular interest to policy makers, given the lack of deep forward markets in many individual commodities, and broad aggregate commodity indices in particular. We also explore the reverse relationship (commodity prices forecasting exchange rates) but find it to be notably less robust. We offer a theoretical resolution, based on the fact that exchange rates are strongly forward-looking, whereas commodity price fluctuations are typically more sensitive to short-term demand imbalances. © 2010 by the President and Fellows of Harvard College and the Massachusetts Institute of Technology.

Accuracy of proactive case finding for mental disorders by community informants in Nepal.

BACKGROUND: Accurate detection of persons in need of mental healthcare is crucial to reduce the treatment gap between psychiatric burden and service use in low- and middle-income (LAMI) countries. AIMS: To evaluate the accuracy of a community-based proactive case-finding strategy (Community Informant Detection Tool, CIDT), involving pictorial vignettes, designed to initiate pathways for mental health treatment in primary care settings. METHOD: Community informants using the CIDT identified screen positive (n = 110) and negative persons (n = 85). Participants were then administered the Composite International Diagnostic Interview (CIDI). RESULTS: The CIDT has a positive predictive value of 0.64 (0.68 for adults only) and a negative predictive value of 0.93 (0.91 for adults only). CONCLUSIONS: The CIDT has promising detection properties for psychiatric caseness. Further research should investigate its potential to increase demand for, and access to, mental health services.

The accuracy and completeness for receipt of colorectal cancer care using Veterans Health Administration administrative data.

BACKGROUND: The National Comprehensive Cancer Network and the American Society of Clinical Oncology have established guidelines for the treatment and surveillance of colorectal cancer (CRC), respectively. Considering these guidelines, an accurate and efficient method is needed to measure receipt of care. METHODS: The accuracy and completeness of Veterans Health Administration (VA) administrative data were assessed by comparing them with data manually abstracted during the Colorectal Cancer Care Collaborative (C4) quality improvement initiative for 618 patients with stage I-III CRC. RESULTS: The VA administrative data contained gender, marital, and birth information for all patients but race information was missing for 62.1% of patients. The percent agreement for demographic variables ranged from 98.1-100%. The kappa statistic for receipt of treatments ranged from 0.21 to 0.60 and there was a 96.9% agreement for the date of surgical resection. The percentage of post-diagnosis surveillance events in C4 also in VA administrative data were 76.0% for colonoscopy, 84.6% for physician visit, and 26.3% for carcinoembryonic antigen (CEA) test. CONCLUSIONS: VA administrative data are accurate and complete for non-race demographic variables, receipt of CRC treatment, colonoscopy, and physician visits; but alternative data sources may be necessary to capture patient race and receipt of CEA tests.