The growth terraces in molecular beam epitaxy-grown AlxGa1-xAs multilayers are observed on the ultrahigh vacuum cleaved (110) cross-sectional plane using scanning tunneling microscopy. Under regular growth conditions on 2-degrees off oriented vicinal surfaces, we observe step bunching of 2-8 atomic layers and a corresponding extension of the terrace length instead of monolayer steps. These results demonstrate that the roughness of quantum confinement layers can be studied down to the atomic scale in a direct way.