Publication - IEEE 2023
Abstract
Stack Overflow and other similar forums are commonly used by developers to seek answers for their software development as well as privacy-related concerns. Recently, ChatGPT has been used as an alternative to generate code or produce responses to developers' questions. In this paper, we aim to understand developers' privacy challenges by evaluating the types of privacy-related questions asked on Stack Overflow. We then conduct a comparative analysis between the accepted responses given by Stack Overflow users and the responses produced by ChatGPT for those extracted questions to identify if ChatGPT could serve as a viable alternative. Our results show that most privacy-related questions are related to choice/consent, aggregation, and identification. Furthermore, our findings illustrate that ChatGPT generates similarly correct responses for about 56% of questions while for the rest of the responses, the answers from Stack Overflow are slightly more accurate than ChatGPT.
Click Here for More Information, Click Here to Read for Free
About
This paper was written with members of the PERC_Lab. Zack Delile led the project, and I was a co-author alongside Joe Godinez, Garrett Engstrom, Theo Brucker, Kenzie Young, and Sepideh Ghanavati.
In our research, we sought to assess whether ChatGPT could serve as a viable alternative to Stack Overflow in answering privacy-related questions and generating similarly correct results. Our research was based on 932 questions extracted between 2016 and 2023 from Stack Overflow by lab memeber Sam Morse. Next, we randomly selected 270 questions and answers from the data set,then we conducted a comparative analysis with the responses received from ChatGPT. Next, we developed an annotation strategy to classify the types of QAs based on two well-known privacy taxonomies. After four rounds of annotations with three groups of two annotators (i.e., six in total), and two rounds of discussions to resolve the differences, we create a dataset of 92 pairs of multi-labeled privacy-related QAs.
We then follow prompt engineering and generate answers for Stack Overflow privacy-related questions in ChatGPT and develop a study to identify whether the solutions from ChatGPT match with the accepted answers or the highest upvoted answers from Stack Overflow. Our results show that when given the same prompt, ChatGPT responses match with Stack Overflow answers in 56.1% of cases. When it does not match, SO is slightly more accurate than ChatGPT. This result indicates that ChatGPT may in the future be used as an alternative to SO and other forums. However, since both of these tools show an accuracy of less than 75%, developers need to use the forums and ChatGPT with caution to ensure compliance with regulations and protect users’ privacy.