We study a fully-discrete stabilizer-free weak Galerkin (SFWG) finite element method for the initial-boundary value problem of variable-order Caputo time-fractional diffusion equation. Optimal convergence order in L2$$ {L}_2 $$ norm for L1$$ L1 $$ fully-discrete SFWG scheme is obtained. Furthermore, we develop a fast evaluation algorithm to improve the computational efficiency. Numerical experiments are performed to verify theoretical results.