Accurate and fast recognition of medical images is a key step for clinical diagnosis. With the rapid development of convolutional neural networks in image processing, the convolutional neural network (CNN) approaches based on the U-shape structure have been used in various medical image recognition projects. However, the context information extraction capability of U-shape structure is insufficient and boundaries of segmentation results are blurred. In this paper, we propose a lightweight contextual and channel fusion network (LCCF-Net) for medical segmentation, which fuses multi-scale context information, preserves channel information and minimizes the amount of computation in the network. LCCF-Net is mainly composed of codec block, Shifted Windows Self-attention Connection (SWSC) block, Xception Group (XG) block and Multiple Pooling (MP) block. We design to input the features of each level of U-net encoder into their own SWSC blocks for self-attention mechanism operation, and then fuse with decoding features to participate in feature reconstruction. The proposed Multiple Pooling (MP) block combines multi-scale context information in high-level features, which can fuse rich context information. In order to reduce the number of parameters in the network, we design XG block instead of conventional convolution. Comprehensive results show that the proposed method outperforms other state-of-the-art methods for kidney tumor recognition, retinal vessel detection and COVID 19 segmentation.