Amorphous polymers are normally isotropic in their physical properties, however, upon stress their structural randomness is disturbed and they become anisotropic. There is a close connection between the optical anisotropy and the elastic (or mechanical) anisotropy, since both are related to the type of symmetry exhibited by the molecular structure. On the origin of Gaussian network theory, a phenomenological constitutive framework was proposed to study the photo-elastic transition and working mechanism of the thermo-/chemo-responsive shape-memory effect (SME) in amorphous shape memory polymers (SMPs). Optically refractive index was initially employed to couple the stress, strain and the anisotropy of the random link in macromolecule chain. Based on the Arrhenius law, a constitutive framework was then applied for the temperature dependence of optical (or elastic or mechanical) anisotropy according to the fictive temperature parameter. Finally, the phenomenological photo-elastic model was proposed to quantitatively identify the influential factors behind the thermo-/chemo-responsive SME in SMPs, of which the shape recovery behavior is predicted and verified by the available experimental data reported in the literature.