Balreet’s paper “Analyzing Developer Use of ChatGPT Generated Code in Open Source GitHub Projects” was accepted for publication at the MSR Mining Challenge 2024! Super congrats Balreet and co-author Wentao! This paper was a collaboration with Dr. Sarah Nadi from New York University Abu Dhabi.
Abstract: “The rapid development of large language models such as ChatGPT have made them particularly useful to developers in generating code snippets for their projects. To understand how ChatGPT’s generated code is leveraged by developers, we conducted an empirical study of 3,044 ChatGPT-generated code snippets integrated within GitHub projects. A median of 54% of the generated lines of code is found in the project’s code and this code typically remains unchanged once added. The modifications of the 76 code snippets that changed in a subsequent commit, consisted of minor functionality changes and code reorganizations that were made within a day. Our findings offer insights that help drive the development of AI-assisted programming tools. We highlight the importance of making changes in ChatGPT code before integrating it into a project.”
A preprint of the paper is available here.