The neocortex is the part of the brain that is involved in perception, cognition, and volitional motor control. In mammals, the neocortex is a highly dynamic structure that has been dramatically altered within an individual’s lifetime and in different lineages throughout the course of evolution. These alterations account for the remarkable variations in behavior that species exhibit. Because we cannot study the evolution of the neocortex directly, we must make inferences about the evolutionary process from a comparative analysis of brains, and study the developmental mechanisms that give rise to alterations in the brain. Comparative studies allow us to appreciate the types of changes that have been made to the neocortex and the similarities that exist across taxa, and ultimately the constraints imposed on the evolving brain. Developmental studies inform us about how phenotypic transitions may arise by alterations in developmental cascades or changes in the physical environment in which the brain develops. We focus on how early experience shapes the functional organization and connectivity of each individual’s brain and behavior to be uniquely optimized for a given sensory milieu. Such plasticity plays an integral role in shaping the brains of normal individuals. Changes in sensory input that occur early in development leads to dramatic changes in both the normal organization and connections of the neocortex as well as in sensory mediated behavior. Studies have also demonstrated that enhanced sensory experience that occurs during critical periods of development has a profound effect on the resultant organization and connectivity of the neocortex. In our experiments, we examined the specific types of alterations that occur when individuals develop with lost or enhanced sensory inputs in both experimental and natural settings. Because all aspects of complex social experience including parental rearing and sibling interactions are mediated by our sensory systems, it follows that these types of complex patterns of sensory inputs are fundamentally important for shaping both the organization and connectivity of the neocortex. In turn, the ultimate behavior generated by the neocortex will be highly adaptive for the context in which the individual develops.