Detecting actions of fruit flies

In this thesis we describe a system that tracks fruit flies in video and automatically detects and classifies their actions. We introduce Caltech Fly-vs-Fly Interactions, a new dataset that contains hours of video showing pairs of fruit flies engaging in social interactions, and is published with complete expert annotations and articulated pose trajectory features. We compare experimentally the value of a frame-level feature representation with the more elaborate notion of bout features that capture the structure within actions. Similarly, we compare a simple sliding window classifier architecture with a more sophisticated structured output architecture, and find that window based detectors outperform the much slower structured counterparts, and approach human performance. In addition we test the top performing detector on the CRIM13 mouse dataset, finding that it matches the performance of the best published method.