This cross-sectional study aimed to investigate the associations between salivary testosterone concentrations and cognitive function in 70-year-old Japanese elderly people without dementia and stroke. Methods: Participants were 197 Japanese community-dwelling people aged 69–71 years. Their salivary samples were collected, and their cognitive function was assessed using the Japanese version of the Montreal Cognitive Assessment (MoCA-J). Participants were also administered a 10-item recall and a 24-item recognition test. The data for 179 (106 men and 73 women) individuals were analyzed, excluding individuals with a past history of stroke and dementia. Multivariate logistic regression analyses were performed after adjusting for lifestyle factors and analyzing data separately for men and women. Results: MoCA-J scores showed that men with low testosterone concentrations had a significantly greater risk of low cognitive performance than those with high testosterone concentrations (adjusted odds ratio: 4.72, 95% confidence interval: 1.06–21.00), while no significant association was found in women. The 10-item recall test scores showed that higher testosterone concentrations were significantly associated with greater recall in the second trial in women (standardized beta = 0.24, P = 0.040), whereas no significant association was found in men. Salivary testosterone concentrations were positively associated with better cognitive performance in older men and women. Conclusions: The associations between salivary testosterone concentrations and cognitive function were shown by different tasks for men and women.