The ability to organize and memorize the unfolding of events over time is a fundamental feature of cognition, which develops concurrently with the maturation of the brain. Nonetheless, how temporal processing evolves across the lifetime as well as the links with the underlying neural substrates remains unclear. Here, we intend to retrace the main developmental stages of brain structure, function, and cognition linked to the emergence of timing abilities. This neurodevelopmental perspective aims to untangle the puzzling trajectory of temporal processing aspects across the lifetime, paving the way to novel neuropsychological assessments and cognitive rehabilitation strategies.