Who Watches the Watchmen: A Security-focused Review on Current State-of-the-art Techniques, Tools, and Methods for Systems and Binary Analysis on Modern Platforms

Malicious software, a threat users face on a daily basis, have evolved from simple bankers based on social engineering to advanced persistent threats. Recent research and discoveries reveal that malware developers have been using a wide range of anti-analysis and evasion techniques, in-memory attacks, and system subversion, including BIOS and hypervisors. In addition, code-reuse attacks like Returned Oriented Programming emerge as highly potential remote code execution threats. To counteract the broadness of malicious codes, distinct techniques and tools have been proposed, such as transparent malware tracers, system-wide debuggers, live forensics tools, and isolated execution rings. In this work, we present a survey on state-of-the-art techniques that detect, mitigate, and analyze the aforementioned attacks. We show approaches based on Hardware Virtual Machines introspection, System Management Mode instrumentation, Hardware Performance Counters, isolated rings (e.g., Software Guard eXtensions), as well as others based on external hardware. We also discuss upcoming threats based on the very same technologies used for defense. Our main goal is to provide the reader with a broader, more comprehensive understanding of recently surfaced tools and techniques aiming at binary analysis for modern platforms.