We report the nonperturbative behavior of the twisted Polyakov loop (TPL) coupling constant for the SU(3) gauge theories defined by the ratio of Polyakov loop correlators in finite volume with twisted boundary condition. We reveal the vacuum structures and the phase structure for the lattice gauge theory with the twisted boundary condition. Carrying out the numerical simulations, we determine the nonperturbative running coupling constant in this renormalization scheme for the quenched QCD and N-f = 12 SU(3) gauge theories. First, we study the quenched QCD theory using the plaquette gauge action. The TPL coupling constant has a fake fixed point in the confinement phase. We discuss this fake fixed point of the TPL scheme and obtain the nonperturbative running coupling constant in the deconfinement phase, where the magnitude of the Polyakov loop shows the nonzero values. We also investigate the system coupled to fundamental fermions. Since we use the naive staggered fermion with the twisted boundary condition in our simulation, only multiples of 12 are allowed for the number of flavors. According to the perturbative two-loop analysis, the N-f = 12 SU(3) gauge theory might have a conformal fixed point in the infrared region. However, recent lattice studies show controversial results for the existence of the fixed point. We point out possible problems in previous work, and present our careful study. Finally, we find the infrared fixed point (IRFP) and discuss the robustness of the nontrivial IRFP of a many-flavor system under the change of the analysis method. Some preliminary results were reported in the proceedings [E. Bilgici et al., PoS(Lattice 2009), 063 (2009); Itou et al., PoS(Lattice 2010), 054 (2010)] and the letter paper [T. Aoyama et al., arXiv: 1109.5806 [hep-lat]]. In this paper we include a review of these results and give a final conclusion for the existence of the IRFP of SU(3) N-f = 12 massless theory using the updated data.