Visual scene understanding requires much more than a list of the objects present in the scene and their locations. To understand a scene, plan action on it, and predict what will happen next we must extract the relationships between objects (e.g., support and attachment), their physical properties (e.g., mass and material), and the forces acting upon them. One view is that we do this with the use of a “mental physics engine” that represents this information and runs forward simulations to predict what will happen next. Over the last several years we have been testing this idea with fMRI. I will review evidence that certain brain regions in the parietal and frontal lobes behave as expected if they implement a mental physics engine: they respond more strongly when deciding about physical than visual properties and when viewing physical versus social stimuli (Fischer et al, 2016), and they contain scenario-invariant information about object mass inferred from motion trajectories (Schwettmann et al, 2019), and about the stability of a configuration of objects (Pramod et al, 2022). In ongoing work led by Pramod RT, we further find evidence that these regions contain information about object contact, a property critical for predicting what will happen next, and most tellingly we find that we can decode predicted future contact from observed current contact, implicating these regions in forward simulation. Another line of work with Vivian Paulun asks whether these brain regions process only the physics of “Things”, or whether they also process “Stuff”. I will argue that these and other findings provide preliminary evidence for a physics engine in the brain, while also discussing the several key predictions yet to be tested to better nail this hypothesis.
