The inferior colliculus (IC) integrates a variety of inputs to perform spectrotemporal processing in the primary auditory pathway, including temporal to rate transformations. The temporal to rate transformations occurring in the IC make it important to understand how the auditory pathway is able to adapt to changes in hearing abilities, such as due to aging or noise-induced hearing loss. In this study, we used consonant-vowel sounds that varied in voice onset time (VOT) (ba to pa), either with tokens or the envelopes of those tokens to modulate a noise carrier. Synchronized neural populations were recorded non-invasively as envelope-following responses (EFRs) in young and aged rats. In addition, local field potentials (LFPs) and unit activities were recorded in the inferior colliculus, enabling some measure of input to output transformation within the IC. We found that both EFRs and LFPs were degraded in older animals, even after compensating for hearing thresholds. However, IC unit activity was similar between young and aged rats in many cases using simple measures such as firing rates. We then tested the related question of whether individual sites or populations were able to discriminate between the different VOT stimuli. A template-matching classification model was generated in which single-trial responses were correlated with aggregate trends. IC units were found to discriminate stimuli above chance but still made errors. Integration over a population of units reduced variability and increased performance. It was found that stimulus discrimination was similar for VOT envelopes modulating a noise carrier, but declined in older animals for the original tokens. These results suggest that there may be multiple mechanisms of compensation to maintain neural representations in older animals, including compensation within the IC.