Structure from Silence:
Learning Scene Structure from Ambient Sound


Ziyang Chen*
Xixi Hu*
Andrew Owens

University of Michigan

CoRL 2021 (Oral)

[OpenReview]
[ArXiv]
[Code] (Coming soon)
[Talk]
[Slides]


What does ambient sound tell us about scene structure?. (a) We collect an "in-the-wild" dataset of paired audio and RGB-D recordings from quiet indoor scenes. (b) Given audio from a scene, we estimate distance to a wall. (c) We use this ambient audio to learn useful representations using multimodal self-supervision.


Abstract

From whirling ceiling fans to ticking clocks, the sounds that we hear subtly vary as we move through a scene. We ask whether these ambient sounds convey information about 3D scene structure and, if so, whether they provide a useful learning signal for multimodal models. To study this, we collect a dataset of paired audio and RGB-D recordings from a variety of quiet indoor scenes. We then train models that estimate the distance to nearby walls, given only audio as input. We also use these recordings to learn multimodal representations through self-supervision, by training a network to associate images with their corresponding sounds. These results suggest that ambient sound conveys a surprising amount of information about scene structure, and that it is a useful signal for learning multimodal features.



Dataset



[Download Link] (coming soon)


Talk




Qualitative Results

Obstacle Detection Demo

Robot Demo




Paper and Supplementary Material

Ziyang Chen*, Xixi Hu*, Andrew Owens.
Structure from Silence: Learning Scene Structure from Ambient Sound.
CoRL 2021.
(OpenReview) (Arxiv)


[Bibtex]


Acknowledgements

We thank James Traer for his invaluable help explaining work on human perception of ambient sound. We would also like to thank Linyi Jin and Shengyi Qian for help with visual experiment setups and their valuable suggestions. We also thank David Fouhey for his comments and feedback, Peter Gaskell for his help on robot equipment, and Yichen Yang for the help on data collection. We thank the valuable suggestions and feedbacks from David Harwath, Kristen Grauman and UT-Austin Computer Vision Group. This work was funded in part by DARPA Semafor and Cisco Systems. The views, opinions and/or findings expressed are those of the authors and should not be interpreted as representing the official views or policies of the Department of Defense or the U.S. Government. The webpage template was originally made by Phillip Isola and Richard Zhang for a Colorization project.