This project received an MICDE Catalyst Grant in summer 2022.



Our proposed Behavior classification pipeline. We will use three steps: (1) Preprocessing our video data by creating a localized video stream using our SOT or MOT pipelines. The example illustrates three tracked animals. (2) Encoding each animal video stream into a latent representation. (3) Classify each animals’ latent representation into our behavioral categories using a classifier. Our 2nd and 3rd steps are jointly optimized using our behavior classification dataset.

Our research team will develop an accessible computer vision toolbox to automatically track multiple animals and classify their behaviors in complex social environments. We will harness state-of-the-art developments in machine learning and computer vision as well as a rich dataset of manually annotated video recordings.

Principal Investigators

Ada Eban-Rothschild, Assistant Professor of Psychology, U-M

Justin Johnson, Assistant Professor of Electrical Engineering and Computer Science, U-M