Objective In practical free-space optical (FSO) communication systems, the channel state information of the receiver is not precise, and this channel estimation error can result in a mismatch during the signal demodulation. Therefore, the system channel state information should be considered non-ideal to more closely match the complexity and uncertainty of the real operating environment for designing an FSO system that meets the practical application scenarios. Methods The system channel state information needs to be considered as non-ideal to more closely match the complexity and uncertainty of the real operating environment and then an FSO system is designed to meet the practical application scenarios. Meanwhile, geometric shaping-quadrature amplitude modulation (GS-QAM) is obtained by optimizing a number of metrics, such as mutual information (MI) and generalized mutual information (GMI), which will relocate the constellation points in the geometric space. We optimize a practical FSO 16QAM communication scheme with adaptive geometric shaping (GS) based on the closed form probability density function (PDF) of the non-ideal channel gain and introduce a trust domain algorithm with a nonlinear conjugate gradient algorithm, which is employed to optimize the position of each constellation point on the constellation map to adapt to the channel conditions. As a result, this can solve the problem of traversing the optimal position of the constellation points with GMI as the objective function. Finally, in the range of the signal-to-noise ratio (SNR), designing the optimal constellation shape can maximize the reachable information rate in optical communication while maintaining a certain bit error rate (B ER). Additionally, the trust domain method is to set a limited range in the neighborhood of the current solution to search within that range. The trust domain model represents the objective function better. It will search for the minimum value of the model within that region as a step size. If a step size is not appropriate, the region is reduced and the minimum point is searched again. As an optimization algorithm that requires only the first order derivatives of the function without reliance on additional parameter inputs, the conjugate gradient method has excellent convergence and stability properties and is adopted to solve unconstrained optimization problems. We combine the trust domain optimization algorithm with the nonlinear conjugate gradient method to improve the amount of GMI as an optimization criterion to optimize the shape of geometric constellations for optical communication systems. Results and Discussions We explore the optimal position traversal problem of constellation points under Gamma-Gamma turbulence channels and non-ideal Gamma-Gamma turbulence channels by employing the trust domain algorithm and the nonlinear conjugate gradient algorithm with GMI as an objective function. Simulation results show that it is possible to achieve adaptive selection of the optimal GS distribution at the transmitter to transmit 16QAM signals in different SNR conditions. In the ideal channel, the GMI performance is significantly improved with the adaptive GS 16QAM scheme. Figure 5 shows that the GMI values of GS are all higher than those of the uniform distribution under fixed turbulence intensity, indicating that the proposed adaptive GS scheme improves the system performance and optimizes the rate loss caused by the uniform distribution. In Fig. 6, under the fixed correlation coefficient, the GMI values of GS are also higher than those of the uniform distribution, indicating that the adaptive 16QAM GS technique has a sound effect of compensating the estimation error of the non-ideal channel. Under ideal channels, the adaptive constellation GS technique introduced in Fig. 8 provides stable NGMI performance for the system, while under non-ideal channels, Fig. 9 shows that the GS technique ensures system reliability. In summary, by combining the conjugate gradient method, the trust domain algorithm, and the constellation GS technique, it is possible to achieve the dynamic optimization of the constellation in a wide range of SNRs and atmospheric turbulence conditions, thus ensuring the stability and high efficiency of the communication link. Conclusions We propose an adaptive GS coded modulation optimization scheme for FSO communication to solve the optimal position traversal problem of constellation points with GS as the objective function by transforming the problem into an unconstrained optimization problem using the GMI gradient. The performance of GMI and NGMI with GS and uniform distribution is analyzed and compared for different turbulence intensities, received SNRs, and non-ideal channel correlation coefficients under ideal and non-ideal channels. When the SNR is kept constant, the turbulence intensity increase introduces random jitter and distortion of the transmitted optical signals, which causes the deviation of the signal constellation points from the original ideal position and makes the constellation points more dispersed. As a result, this affects the judgment accuracy of the signals at the receiving end and ultimately increases the BER. The smaller turbulence intensity or larger SNR leads to more compact constellation point layouts after GS. The NGMI value increases with the rising non-ideal channel correlation coefficient, and the larger correlation coefficient makes it closer to the ideal channel conditions. Additionally, the difference between the NGMI values of different correlation coefficients decreases with the increasing SNR. This is due to the fact that the distribution of constellation points is optimized by GS, which enhances the signal's immunity to noise, adapts and optimizes the signal judgment region, and enables the system to maintain a high communication quality in non-ideal atmospheric conditions. Therefore, the system NGMI values are above the NGMI threshold after adopting adaptive constellation GS, which means that the introduction of adaptive constellation GS technique plays a vital role in providing stable NGMI performance.
