On maximum residual block and two-step Gauss-Seidel algorithms for linear least-squares problems

The block Gauss-Seidel algorithm can significantly outperform the simple randomized Gauss-Seidel algorithm for solving overdetermined least-squares problems since it moves a large block of columns rather than a single column into working memory. Here, with the help of the maximum residual rule, we construct a two-step Gauss-Seidel (2SGS) algorithm, which selects two different columns simultaneously at each iteration. As a natural extension of the 2SGS algorithm, we further propose a multi-step Gauss-Seidel algorithm, that is, the maximum residual block Gauss-Seidel (MRBGS) algorithm for solving overdetermined least-squares problems. We prove that these two different algorithms converge to the unique solution of the overdetermined least-squares problem when its coefficient matrix is of full column rank. Numerical experiments on Gaussian models as well as on 2D image reconstruction problems, show that 2SGS is more effective than the greedy randomized Gauss-Seidel algorithm, and MRBGS apparently outperforms both the greedy and randomized block Gauss-Seidel algorithms in terms of numerical performance.