We use Raman scattering to study the composition of InAs self-assembled quantum dots (QDs) embedded in an AlxGa1-xAs matrix. When Al is introduced into the matrix (0.15 < x < 0.4), the QD phonon frequency exhibits a downward frequency shift with respect to x=0. This shift is attributed to the incorporation of Al into the QDs. For the samples with an Al-rich matrix (x greater than or similar to 0.6), the QD phonon frequency is higher than that for the samples with a Ga-rich matrix, which suggests that Al/In intermixing is weaker than Ga/In intermixing. These processes affect strongly the energy of the QD luminescence.