The current X-ray prohibited item detection system needs to further improve its detection accuracy in complex backgrounds, and also faces challenges in terms of real-time detection. Aiming at the problems of poor image contrast, obscuring object localization, and low detection efficiency, this paper proposes an x-ray prohibited item detection method based on inverted residual layer aggregation IRLAN and lightweight contextual downsampling LCDown. Firstly, FasterNet is adopted as the backbone network of the model to optimize feature representation and improve detection speed. Secondly, the inverted residual layer aggregation strategy and lightweight context downsampling strategy are proposed to enhance the model's feature fusion ability and to solve the problem of unclear edges and details of the items due to low contrast and noise that usually exists in X-ray images. Finally, the efficient intersection over union loss function is introduced to improve the ability of prohibited item localization and accelerate the convergence speed of the model. Compared with the RT-DETR-R50 algorithm, which has the highest accuracy among the current target detection algorithms, the mAP50 of the proposed method on the SIXray and OPIXray datasets are improved by 1.4% and 1.3%; meanwhile, the FPS reaches 42.9 frames per second, which can meet the requirement of real-time detection. Experimental results with Grad-CAM feature heatmaps show that the proposed method achieves good detection results on public X-ray image datasets.