Efficiency analysis on a truncated Newton method with preconditioned conjugate gradient technique for optimization

It has been shown by a large amount of numerical experiments that among the local algorithms for solving unconstrained optimization problems, the truncated Newton method with preconditioned conjugate gradient (PCG) subiterations is very efficient. In this paper, we investigate its efficiency from theoretical point of view. The question is, compared with Newton's method with Cholesky factorization, how much it is more efficient in theory. We give a quantitative answer by constructing a truncated Newton method with PCG subiterations -- Algorithm II below. Suppose Newton's method is convergent with a one-step, Q-order alpha rate (alpha greater than or equal to 2). We first prove that Algorithm II has the same convergence rate. We then study its average Dumber of arithmetic operations per step and the corresponding number which Newton's method needs. An upper bound for the ratio of these two numbers is obtained. This upper bound is a quantitative estimate of the saving which Algorithm II can achieve from theoretical point of view. Its values, which are listed in the paper, show that the saving is rather remarkable.