We consider the radiative transfer equation (RTE) with two scalings in this paper: one is the diffusive scaling whose macroscopic limit is a diffusion equation, and the other is a highly forward peaked scaling, wherein the scattering term is approximated by a Fokker-Planck operator as a limit. In the inverse setting, we are concerned with reconstructing the scattering and absorption coefficients using boundary measurements. As the measurement is often polluted by errors, both experimental and computational, an important question is to quantify how the error is amplified or suppressed in the process of reconstruction. Since the solution to the forward RTE behaves differently in different regimes, it is expected that stability of the inverse problem will vary accordingly. Particularly, we adopted the linearized approach and showed, in the former case, that the stability degrades when the limit is taken, following a similar approach as in [K. Chen, Q. Li and L. Wang, Inverse Problems, 34 (2018), 025004]. In the latter case, we showed that a full recovery of the scattering coefficient is less possible in the limit.