It has been suggested that the acquisition of object labels proceeds in a bottom-up fashion during infancy: labels initially act as additional features of the associated objects, then they become referential signals denoting correlated sets of visual features, and are eventually construed as standing for concepts instantiated in objects. In contrast to these claims, we find in our lab that, at least in ostensive contexts, infants around one year of age (i) preferentially map labels onto the referent of concurrent communicative gestures, (ii) group a set of objects together by a common label even in the absence of shared visual features, and (iii) prefer to learn a new label to a concept they already possess rather than mapping it to an object defined by visual features. Together these findings support the view that mapping words onto concepts is not a late achievement but an expectation that guides the acquisition of words from early on.