Deception Detection System with Joint Cross-Attention

In recent research, the field of biometrics has turned its attention towards detecting deception. Building on insights from criminal psychology, which highlights the significance of facial expressions and voice tones in uncovering deceit, this study introduces a novel system for detecting deception. This system integrates a trio of components: displacement of 68 facial landmarks, action units (AUs), and audio emotion units (EUs). It leverages criminal psychology's findings to track changes in facial expressions across consecutive frames and to assess shifts in emotions through audio analysis. A key discovery is the potential of the facial tensor, derived from the changes in the 68 facial landmarks across frames, as a robust indicator for deception detection. The system's effectiveness is evaluated using three datasets: the public datasets Real-life and Bag-of-lies, and a private dataset, MSPL-YTD. Overall, this new approach shows promise as an effective tool for intelligent deception detection.