Bi-Source Reconstruction-Based Classification Network for Face Forgery Video Detection

Current methods for detecting deep fakes concentrate on specific patterns of forgery like noise characteristics, local textures, or frequency statistics. These approaches assume training and test sets exhibit similar data distributions, which bring severe performance drops and further limit broader applications when migrating unseen domains. Existing works show that reconstruction learning is effective in capturing unseen forgery clues. However, 2D reconstruction is insufficient and can not handle non-frontal face reconstruction, while 3D reconstruction provides more critical details of facial structure and finds accurate forgery regions. In this paper, we propose a bi-source reconstruction based classification network (BRCNet) to incorporate 2D and 3D reconstruction as the supervisions and learn the optimal feature representation. In detail, we employ an encoder-decoder architecture to facilitate reconstruction learning, enhancing the learned representations to detect forgery patterns that are unknown. To further capture forgery evidence across multiple scales, instead of using encoder features from the reconstruction network only, we build a feature improvement network to combine feature details from encoder and decoder features in a multi-scale fashion. In addition, we use the reconstruction difference to supervise the feature aggregation, which enables detecting the subtle and trivial discrepancies between fake and real video frames. Extensive experiments are conducted to validate the performance of our proposed method on several deep fake benchmarks. The results demonstrate the efficacy of our approach, offering promising results and showcasing its potential for practical applications. The source code is available at https://github.com/cccvl/BRCNet.