Grounding Spoken Words in Unlabeled Video Angie Boggust, Kartik Audhkhasi, Dhiraj Joshi, David Harwath, Samuel Thomas, Rogerio Feris, Dan Gutfreund, Yang Zhang, Antonio Torralba, Michael Picheny, James Glass CVPR 2019 Sight and Sound Workshop

Examples of the unsupervised model’s semantic correlation on videos from the YouCook2 dataset. The highlighted regions of each image indicate pixels the model believes are strongly related to the audio below.


In this paper, we explore deep learning models that learn joint multi-modal embeddings in videos where the audio and visual streams are loosely synchronized. Specifically, we consider cooking show videos from the YouCook2 dataset and a subset of the YouTube-8M dataset. We introduce varying levels of supervision into the learning process to guide the sampling of audio-visual pairs for training the models. This includes (1) a fully-unsupervised approach that samples audio-visual segments uniformly from an entire video, and (2) sampling audio-visual segments using weak supervision from off-the-shelf automatic speech and visual recognition systems. Although these models are preliminary, even with no supervision they are capable of learning cross-modal correlations, and with weak supervision we see significant amounts of cross-modal learning.

    title={Grounding Spoken Words in Unlabeled Video},
    author={Boggust, Angie and Audhkhasi, Kartik and Joshi, Dhiraj and Harwath, David and Thomas, Samuel and Feris,
            Rogerio and Gutfreund, Dan and Zhang, Yang and Torralba, Antonio and Picheny, Michael and others},
    booktitle={Proceedings of the IEEE Conference on Computer Vision and Pattern Recognition Workshops},