Purpose Adjuvant endocrine therapy (AET) reduces breast cancer morbidity and mortality, yet women often report suboptimal adherence. Though correlates of AET adherence are well-documented, few studies examine the relative importance of multi-level factors associated with adherence. The aim of this study was to identify factors most strongly associated with AET adherence in women with breast cancer. Methods Between 10/2019 and 6/2021, women (N = 100) with non-metastatic, hormone receptor-positive breast cancer, taking AET who reported AET-related distress enrolled into a clinical trial. Participants completed baseline measures, including the Medication Adherence Rating Scale-5, sociodemographics, and validated measures of anxiety, depression, medication-taking self-efficacy, social support, and treatment satisfaction. We created a latent factor and tested associations between sociodemographic, medical, and psychosocial characteristics and adherence. Associated predictors (p < .10) were entered into a structural model, which was corroborated via multivariate regression modeling. Results A four-indicator latent adherence factor demonstrated good model fit. Participants (M-age = 56.1 years, 91% White) who were unemployed (B = 0.27, SE = 0.13, p = .046) and reported greater treatment convenience (B = 0.01, SE = 0.01, p = .046) reported greater adherence. Scores of participants who reported greater medication-taking self-efficacy (p = .097) and social support (p = .062) approached better adherence. Greater medication-taking self-efficacy (B = 0.08, SE = 0.02, p < .001) and being unemployed (B = 0.28, SE = .14, p = .042) were most strongly associated with greater adherence, inde-pendent of other predictors. Multivariate modeling confirmed similar findings. Conclusions Medication-taking self-efficacy and employment status were associated with AET adherence above other related factors. Implications for Cancer Survivors Enhancing patients' confidence in their ability to take AET for breast cancer may represent an important intervention target to boost adherence