Properties and comparison of estimation methods in a log-linear generalized linear mixed model

Generalized linear mixed models have become a popular choice for modeling correlated and non-normal response data, with an increasing number of methods available for fitting these models. However, due to the complexities of the likelihood functions, limited work has been carried out to examine the asymptotic behavior and performance of the methods, including the effects of small sample size, large variance component values and stability of the computational algorithms. In this article, we propose a choice of parameter estimation within an iterative bias correction method developed by Kuk [1995, Asymptotically unbiased estimation in generalized linear models with random effects. Journal of the Royal Statistical Society, Series B, 57, 395-407.] for fitting generalized linear mixed models to improve computational efficiency. We investigate the performance of a commonly used method, maximum likelihood, along with the iterative bias correction method and penalized quasi-likelihood in the fitting of a log-linear generalized linear mixed model with an autoregressive correlation structure for the random effects through the use of extensive simulation studies. In addition, the robustness of the methods for non-normally distributed random effects is examined.