The formation of nanoscale self-assembled compositional patterns in monolayer bulk-immiscible alloy films is studied from first principles within the framework of a previously proposed hybrid atomistic-continuum model [V. Ozolins et al., Phys. Rev. Lett. 88, 096101 (2002)]. The details surrounding the parametrization of the model from first-principles calculations are described for both hexagonal (0001) and bee (110) substrates, and we demonstrate how the theoretical model can be employed in Monte Carlo simulations as a predictive framework for modeling the structure and finite-temperature stability of compositional patterns. The methodology is applied in a comparative study of equiatomic FeAg pseudornorphic alloy films on Mo(110) and Ru(0001) substrates. Stripe patterns with periodicities of a few nanometers are predicted to be stable in both systems, which is in good agreement with available experimental data for FeAg/Mo(110). The regularity of the stripe patterns and their stability with respect to disordering are found to be substantially enhanced on the anisotropic Mo(110) substrate relative to the nearly isotropic Ru(0001) surface, despite the slightly stronger ordering energetics in the latter system. A comparison of the results of the present study to the predictions of continuum theories commonly employed to describe pattern formation on crystalline surfaces serves to highlight the limitations of such models in the application to patterns with periodicities with length scales of approximately ten atomic spacings.