Cognitive impairment has emerged as a critical global public health issue. This study investigated the association between community and residential environments and the risk of cognitive impairment among Chinese middle-aged and older adults. The study included a final sample of 10,471 participants from the 2011–2020 The China Health and Retirement Longitudinal Study (CHARLS) cohort. Community and residential environment, and cognitive impairment, were assessed using several cognitive tests. The analyzed using the Cox Proportional Hazards Mode adjusting for age, gender, education level, marital status, residence areas, Body Mass Index (BMI), smoking status, alcohol consumption and chronic disease status. After adjusting for all Covariates, participants residing in relaxed community-based residential environments exhibited a 3.8% increased risk of cognitive impairment compared to those in crowded community environments (HR = 1.038). Middle-aged and older adults with access to garbage disposal services had a 19.3% reduced risk of cognitive impairment (HR = 0.807). Community environments featuring age-friendly facilities, such as open-air fitness equipment (HR = 0.834) and accessibility for handicapped individuals (HR = 0.948), were associated with a 16.6% and 5.2% reduction in the risk of cognitive impairment, respectively. In community social support environments, having a social organization for the older adults was associated with a 16.2% reduction in cognitive impairment risk among middle-aged and older adults (HR = 0.838). Regarding housing factors, concrete and steel buildings were associated with an 18.7% lower risk of cognitive impairment compared to adobe or other types of buildings (HR = 0.813). Domestic amenities, including running water supply (HR = 0.847), broadband connectivity (HR = 0.650), and use of clean energy (HR = 0.957), were associated with a 15.3%, 35%, and 4.3% reduction in the risk of cognitive impairment, respectively. Additionally, indoor tidiness (HR = 0.875) was associated with a 12.5% reduction in cognitive impairment risk. The good community and residential environment are correlated with the lower risk of cognitive impairment in middle-aged and older Chinese adults. Specifically, for the prevention and control of cognitive impairment, community and residential planning and construction should prioritize both the basic social and environmental factors and the internal built environment.