Keras-tensorflow code for training a frame-by-frame binary classifier with video input + code for computing targets.
Please see the blog posts for a detailed description of this project. https://m-lin-dm.github.io/ConvLSTM_Fish_1/
YouTube video presenting my results:
Project In Brief - TLDR
- Under my laboratory conditions, Serpae tetra fish exhibit aperiodic, "bursty" movement patterns---typically, one fish begins chasing another when the two get too close together. [video]
- I hypothesized that ConvLSTM would be particularly well-suited to model these dynamics. The underlying assumption is that the fishes' prior dynamics encodes some information on if/when a burst is likely to occur.
- Research Goal: Train a ConvLSTM-based model that can take a video clip of fish as input and predict whether or not a burst of motion will occur during the subsequent 1.5 seconds. (Visually, this burst corresponds to sudden aggressive interactions between two or more fish and a spike in group mean speed.)
- Methods: I first filmed a group of 6 Serpae tetra fish from overhead, for 30 mins. I converted this into an image dataset containing ~54000 grayscale frames with 120x160x1 resolution. I trained a ConvLSTM-based model to act as a frame-by-frame binary classifier. This recurrent neural network takes as input a 4 second clip of video frames (stacked as a single 4D tensor of shape (time-steps x width x height x channels)) and outputs the probability that a 'spike' in group mean velocity will occur during the next 1.5 seconds (45 frames).
- Result: The model does fairly well at identifying frames that come right before the fish chase each other. It achieves an ROC-AUC of 0.83 and a peak accuracy of 84.7%, matching the algorithmic, baseline classifier (AUC: 0.84 Peak accuracy: 85.4%). Using a probability threshold that maximizes Youden's statistic, the model achieved a true positive rate of 82.9%, with a false positive rate of 31.0%.
- This 5-part series is a hybrid between an academic paper and an informal guide to doing a deep learning research project. I will
- Introduce the problem we're trying to solve and explain why deep learning/ConvLSTMs constitute a novel approach to it. (This post)
- Report my methods and findings in detail
- Explain key parts of my python-keras-tensorflow code.
- Provide a step-by-step guide outlining the subtasks required to take a project like this from start to finish.