New image encryption algorithm based on hyperchaotic 3D-IHAL and a hybrid cryptosystem

The performance of chaotic systems in generating signals is an essential factor in determining the security of cryptosystems. A new 3D chaotic system is designed based on Hénon and logistic mapping, called 3D Improve-Hénon and Logistic (3D-IHAL), to design a secure and reliable cryptosystem. The attractor phase diagram, Lyapunov exponent, and bifurcation diagram are used to evaluate the performance of 3D-IHAL. NIST tests the randomness of the sequences generated by 3D-IHAL. The results show that 3D-IHAL is suitable for cryptography because of its good dynamic characteristics. In the cryptosystem, the parameters and initial values of 3D-IHAL are generated by the Hash function, and 3D-IHAL generates the key stream required for the encryption phase. In the encryption phase, a hybrid cryptosystem is designed by two different encryption methods, mod and xor, which are used according to the position of the pixel value. Our algorithm is evaluated in public datasets. The evaluation results show that the horizontal, Queryvertical, and diagonal correlations of ciphertext are 0.00012, -0.00089, and -0.00055, respectively, and the information entropy is 7.9992. Compared with the existing algorithms, the proposed algorithm has better security.