Transformer in optronic neural networks for image classification

Transformer, with its self-attention mechanism, can acquire global features while parallelizing training. This study proposes an Optronic Vision Transformer (OPViT), which utilizes spatial light modulators (SLMs) and lens group to realize matrix multiplication. This is the first time that Transformer has been introduced into optical neural network, fully exploiting the advantages of photonic computing to minimize its computational cost. Based on the advantages of OPViT, it is reconfigurable, scalable, and has low space complexity. We achieve excellent classification results directly with only one layer of Transformer and two or three layers of optical convolution, achieving 98.70% and 88.93% test accuracy on the MNIST and Fashion-MNIST datasets, respectively. The classification results outperform existing optical convolutional neural networks, and are even comparable to previous optical architectures connected to electronic networks.