Cell size and individual growth rates vary substantially across genetically identical cell populations. This variation cannot entirely be explained by asynchronous cell division cycles, but also needs to take into account the differences in the histories that cells experience during their lifespan. We describe a stochastic framework to characterise cell size histories in an exponentially growing population. We show that these histories differ from cells observed in isolation, such as observed in mother machines. Quantifying these historical fluctuations allows us to predict the population growth rate. We highlight that the maximum attainable population growth cannot exceed the rate at which an average cell grows, but the population doubles faster than an average cell doubles its size, a prediction we validate using recent single-cell data. For stochastic gene expression, we show that the statistics of cell histories capture the cell-to-cell variability in the growing population and hence they provide us with an ergodic principle between single cells and the population. The theory thus gives fundamental limits on fitness and yields a new interpretation to population snapshot data.