This study aimed to evaluate cognitively stimulating environments throughout life and to examine direct and indirect associations between these environments and late-life cognition. Early-life education, three domains of work complexity (data, people, and things) based on the longest-held occupation, and engagement in late-life leisure activities (LAs) were assessed. A structural equation model was developed using cross-sectional data of 1721 Japanese older adults in 70 ± 1 and 80 ± 1 years. The model confirmed significant direct effects of work complexity with data and late-life LAs on late-life cognitive performance. The associations of education and work complexity with late-life cognition were mediated through the subsequent environment(s). However, the total effects of work complexity with people and things on late-life cognition were insignificant. The findings suggest that cognitively stimulating activities in adulthood and beyond may lead to individual differences in late-life global cognition. In addition, antecedent complex environments might make subsequent life environments more cognitively stimulating. The results are discussed from the perspectives of cognitive plasticity and environmental complexity.