Mynodbcsv: Lightweight Zero-Config Database Solution for Handling Very Large CSV Files.

Volumes of data used in science and industry are growing rapidly. When researchers face the challenge of analyzing them, their format is often the first obstacle. Lack of standardized ways of exploring different data layouts requires an effort each time to solve the problem from scratch. Possibility to access data in a rich, uniform manner, e.g. using Structured Query Language (SQL) would offer expressiveness and user-friendliness. Comma-separated values (CSV) are one of the most common data storage formats. Despite its simplicity, with growing file size handling it becomes non-trivial. Importing CSVs into existing databases is time-consuming and troublesome, or even impossible if its horizontal dimension reaches thousands of columns. Most databases are optimized for handling large number of rows rather than columns, therefore, performance for datasets with non-typical layouts is often unacceptable. Other challenges include schema creation, updates and repeated data imports. To address the above-mentioned problems, I present a system for accessing very large CSV-based datasets by means of SQL. It's characterized by: "no copy" approach - data stay mostly in the CSV files; "zero configuration" - no need to specify database schema; written in C++, with boost [1], SQLite [2] and Qt [3], doesn't require installation and has very small size; query rewriting, dynamic creation of indices for appropriate columns and static data retrieval directly from CSV files ensure efficient plan execution; effortless support for millions of columns; due to per-value typing, using mixed text/numbers data is easy; very simple network protocol provides efficient interface for MATLAB and reduces implementation time for other languages. The software is available as freeware along with educational videos on its website [4]. It doesn't need any prerequisites to run, as all of the libraries are included in the distribution package. I test it against existing database solutions using a battery of benchmarks and discuss the results.

Experimental analysis and computer prediction of CTF/NFI transcription factor DNA binding sites.

Accurate prediction of transcription factor binding sites is needed to unravel the function and regulation of genes discovered in genome sequencing projects. To evaluate current computer prediction tools, we have begun a systematic study of the sequence-specific DNA-binding of a transcription factor belonging to the CTF/NFI family. Using a systematic collection of rationally designed oligonucleotides combined with an in vitro DNA binding assay, we found that the sequence specificity of this protein cannot be represented by a simple consensus sequence or weight matrix. For instance, CTF/NFI uses a flexible DNA binding mode that allows for variations of the binding site length. From the experimental data, we derived a novel prediction method using a generalised profile as a binding site predictor. Experimental evaluation of the generalised profile indicated that it accurately predicts the binding affinity of the transcription factor to natural or synthetic DNA sequences. Furthermore, the in vitro measured binding affinities of a subset of oligonucleotides were found to correlate with their transcriptional activities in transfected cells. The combined computational-experimental approach exemplified in this work thus resulted in an accurate prediction method for CTF/NFI binding sites potentially functioning as regulatory regions in vivo.

Data Acquisition and Computer Plotting of Delamtect Data, HR-269, 1985

The overall system is designed to permit automatic collection of delamination field data for bridge decks. In addition to measuring and recording the data in the field, the system provides for transferring the recorded data to a personal computer for processing and plotting. This permits rapid turnaround from data collection to a finished plot of the results in a fraction of the time previously required for manual analysis of the analog data captured on a strip chart recorder. In normal operation the Delamtect provides an analog voltage for each of two channels which is proportional to the extent of any delamination. These voltages are recorded on a strip chart for later visual analysis. An event marker voltage, produced by a momentary push button on the handle, is also provided by the Delamtect and recorded on a third channel of the analog recorder.

Consumer Advisory: The Drive to Destroy: Removing data from computer hard drives, storage devices & wireless phones, June 2014

The Attorney General’s Consumer Protection Division receives hundreds of calls and consumer complaints every year. Follow these tips to avoid unexpected expense and disappointments. This record is about: The Drive to Destroy: Removing data from computer hard drives, storage devices & wireless phones

Micro-Computer Based Linear Scheduling Application for Highway Construction Project Control, HR-339A, 1995

This report describes a new approach to the problem of scheduling highway construction type projects. The technique can accurately model linear activities and identify the controlling activity path on a linear schedule. Current scheduling practices are unable to accomplish these two tasks with any accuracy for linear activities, leaving planners and manager suspicious of the information they provide. Basic linear scheduling is not a new technique, and many attempts have been made to apply it to various types of work in the past. However, the technique has never been widely used because of the lack of an analytical approach to activity relationships and development of an analytical approach to determining controlling activities. The Linear Scheduling Model (LSM) developed in this report, completes the linear scheduling technique by adding to linear scheduling all of the analytical capabilities, including computer applications, present in CPM scheduling today. The LSM has tremendous potential, and will likely have a significant impact on the way linear construction is scheduled in the future.

Electronic Bulletin Board System, HR-345, 1998

The primary objectives of the Electronic Bulletin Board System (BBS) project were to: (1) Provide an electronic communication tool which would link city and county engineering offices to each other and to other governmental agencies for messaging and data sharing; (2) Provide a dial-up site for reference information or files accessible on-demand; and (3) Provide a "stepping stone" to the world of electronic data transfer, recognizing that most local government employees face a huge complex of technology with limited knowledge of computers and communications tools. The system was designed to be as simple as possible, and to require minimal equipment and software cost to the users. The original system was an Apex 386/25 computer with MS-DOS 5.0 software and the final configuration was an HP Vectra XM Pentium 90 with MS-NT 3.51 and Mustang - Wildcat 5.0 software. The users of the BBS were county engineers and their staff, offices in the central office of the Iowa Department of Transportation (DOT) and Resident Construction Engineers at the Iowa DOT. Much of the activity was between the county engineers, and their staffs, and the Iowa DOT offices with which they have ongoing business activities. The BBS contained files for mapping, Internet e-mail service, Accident Location Analysis System (ALAS) data, Iowa DOT bid lettings, and Autocad and Intergraph maps and standards. The 800 line calls were recorded and gave the best indication of the usage and the trends that were being followed. The usage tended to be higher in the winter months when design activities are occurring and lower in the summer months when the construction is in progress. The project was judged a success. The BBS did provide a "stepping stone" to the world of electronic data transfer.

A Computer Based Information System for County Equipment Cost Records, Interim Report, HR-173, 1975

This report describes the work accomplished to date on research project HR-173, A Computer Based Information System for County Equipment Cost Records, and presents the initial design for this system. The specific topics discussed here are findings from the analysis of information needs, the system specifications developed from these findings, and the proposed system design based upon the system specifications. The initial system design will include tentative input designs for capturing input data, output designs to show the output formats and the items to be output for use in decision making, file design showing the organization of information to be kept on each piece of equipment in the computer data file, and general system design explaining how the entire system will operate. The Steering Committee appointed by Iowa Highway Research Board is asked to study this report, make appropriate suggestions, and give approval to the proposed design subject to any suggestions made. This approval will permit the designer to proceed promptly with the development of the computer program implementation phase of the design.

A Computer Based Information System for County Equipment Cost Records, Final Report, HR-173, 1975

This appendix is divided into three sections. The first section contains abstracts of each of the eight computer programs in the system, instructions for keypunching the three input documents, and computer operating instructions pertaining to each program. The second section contains system flowcharts for the entire system as well as program flowcharts for each program. The last section contains PL/l program listings of each program.

Consumer Advisory: How to Hinder Hackers? Protect Your Password! Tips to better secure your computer and online accounts, September 2014

The Attorney General’s Consumer Protection Division receives hundreds of calls and consumer complaints every year. Follow these tips to avoid unexpected expense and disappointments. This record is about: How to Hinder Hackers? Protect Your Password! Tips to better secure your computer and online accounts