Background The purpose of this study was to investigate whether histogram analysis of an apparent diffusion coefficient (ADC) can serve as a prognostic biomarker for esophageal squamous cell carcinoma (ESCC). Methods This retrospective study enrolled 116 patients with ESCC who received curative surgery from 2006 to 2015 (including 70 patients who received neoadjuvant chemotherapy). Diffusion-weighted magnetic resonance imaging (DWI) was performed prior to treatment. The ADC maps were generated by DWIs at b = 0 and 1000 (s/mm(2)), and analyzed to obtain ADC histogram-derived parameters (mean ADC, kurtosis, and skewness) of the primary tumor. Associations of these parameters with pathological features were analyzed, and Cox regression and Kaplan-Meier analyses were performed to compare these parameters with recurrence-free survival (RFS) and disease-specific survival (DSS). Results Kurtosis was significantly higher in tumors with lymphatic invasion (p = 0.005) with respect to the associations with pathological features. In univariate Cox regression analysis, tumor depth, lymph node status, mean ADC, and kurtosis were significantly correlated with RFS (p = 0.047, p < 0.001, p = 0.037, and p < 0.001, respectively), while lymph node status and kurtosis were also correlated with DSS (p = 0.002 and p = 0.017, respectively). Furthermore, multivariate analysis demonstrated that kurtosis was the independent prognostic factor for both RFS and DSS (p < 0.001 and p = 0.015, respectively). In Kaplan-Meier analysis, patients with higher kurtosis tumors (> 3.24) showed a significantly worse RFS and DFS (p < 0.001 and p = 0.006, respectively). Conclusions Histogram analysis of ADC may serve as a useful biomarker for ESCC, reflecting pathological features and prognosis.