A Multimethod Study of Internet of Things Systems Testing in Industry

As the Internet of Things (IoT) grows, its failures may have dramatic consequences on the lives of people who depend on it. Yet, it is hard to test IoT systems before they are deployed. Several researchers have provided state-of-the-art approaches for testing IoT systems. However, many of those approaches are based on academia rather than industry. Therefore, we conducted a multimethod study of IoT systems testing in the industry with IoT practitioners. We used three methods: 1) an industry survey; 2) practitioners interviews; and 3) analysis of Eclipse IoT surveys. This study focuses on testing IoT systems by industry practitioners. The findings show the following. 1) Testing focuses more on the device, network, and application layers. IoT testing gives more importance to integration testing than acceptance testing. Test coverage is the most important metric, but metrics may vary depending on the project. 2) IoT system testing mainly uses the model-based approach and is often manual or semi-automated, with low adoption of white box testing. Node-RED is commonly used in testing IoT systems, while Amazon AWS IoT is popular for cloud platform testing of IoT devices. 3) Log analysis is the main approach to analyzing the root cause of bugs. 4) The main challenges in IoT testing include the lack of standards, security, connectivity, and reference architecture. Generating test cases and establishing a standard test approach are recommended for further research. This study's findings can help IoT practitioners and researchers to identify and tackle challenges in IoT system testing, leading to future research opportunities.