Mynodbcsv: Lightweight Zero-Config Database Solution for Handling Very Large CSV Files.

Volumes of data used in science and industry are growing rapidly. When researchers face the challenge of analyzing them, their format is often the first obstacle. Lack of standardized ways of exploring different data layouts requires an effort each time to solve the problem from scratch. Possibility to access data in a rich, uniform manner, e.g. using Structured Query Language (SQL) would offer expressiveness and user-friendliness. Comma-separated values (CSV) are one of the most common data storage formats. Despite its simplicity, with growing file size handling it becomes non-trivial. Importing CSVs into existing databases is time-consuming and troublesome, or even impossible if its horizontal dimension reaches thousands of columns. Most databases are optimized for handling large number of rows rather than columns, therefore, performance for datasets with non-typical layouts is often unacceptable. Other challenges include schema creation, updates and repeated data imports. To address the above-mentioned problems, I present a system for accessing very large CSV-based datasets by means of SQL. It's characterized by: "no copy" approach - data stay mostly in the CSV files; "zero configuration" - no need to specify database schema; written in C++, with boost [1], SQLite [2] and Qt [3], doesn't require installation and has very small size; query rewriting, dynamic creation of indices for appropriate columns and static data retrieval directly from CSV files ensure efficient plan execution; effortless support for millions of columns; due to per-value typing, using mixed text/numbers data is easy; very simple network protocol provides efficient interface for MATLAB and reduces implementation time for other languages. The software is available as freeware along with educational videos on its website [4]. It doesn't need any prerequisites to run, as all of the libraries are included in the distribution package. I test it against existing database solutions using a battery of benchmarks and discuss the results.

Assessing small non-zero perceptions of chance: The case of H1N1 (swine) flu risks

Feelings of invulnerability, seen in judgments of 0% risk, can reflect misunderstandings of risk and risk behaviors, suggesting increased need for risk communication. However, judgments of 0% risk may be given by individuals who feel invulnerable, and by individuals who are rounding from small non-zero probabilities. We examined the effect of allowing participants to give more precise responses in the 0-1% range on the validity of reported probability judgments. Participants assessed probabilities for getting H1N1 influenza and dying from it conditional on infection, using a 0-100% visual linear scale. Those responding in the 0-1% range received a follow-up question with more options in that range. This two-step procedure reduced the use of 0% and increased the resolution of responses in the 0-1% range. Moreover, revised probability responses improved predictions of attitudes and self-reported behaviors. Hence, our two-step procedure allows for more precise and more valid measurement of perceived invulnerability. [Authors]