Question Answering Models for Privacy Policies of Mobile Apps: Are We There Yet?

Increasing interest in assisting users by accurately and automatically answering their questions has spawned numerous efforts on using Deep Learning (DL) to solve Question Answering (QA) problems. These efforts have resulted in different QA models capable of answering questions on various topics. Research shows that existing QA models perform well on generic questions (e.g., about sports, news, and common knowledge); however, the ability of QA models to answer basic but important questions about mobile apps' privacy policies is unknown. In this paper, we first derived a set of 43 basic questions concerning mobile apps' privacy policies based on global privacy laws. Then, we constructed two datasets of labeled (Passage, Question, Answer) 3-tuples. Finally, we conducted two main experiments to evaluate the ability and effectiveness of existing QA models in answering basic questions about mobile apps' privacy policies. Our experimental results show that (1) existing QA models perform poorly in answering basic questions about mobile apps' privacy policies, and (2) fine-tuning existing QA models using our QA datasets can help improve their performance. We hope our datasets and findings can help researchers build better QA models for more accurately answering questions about mobile apps' privacy policies.