GitHub Copilot boosts code quality for experienced devs

A recent study conducted by the GitHub Research team evaluates the impact of the AI-based coding tool, GitHub Copilot, on the quality of code written by experienced developers.

GitHub's research team conducted a study involving 202 developers, each with a minimum of five years' experience, to assess the influence of GitHub Copilot on various aspects of code quality. The study found that code produced with the aid of GitHub Copilot was significantly more functional and easier to read, as well as having higher approval rates.

Developers using GitHub Copilot had a 56% greater likelihood of their code passing all unit tests compared to those not using the tool, indicating a significant enhancement in functionality. "Increased functionality: developers with GitHub Copilot access had a 56% greater likelihood of passing all 10 unit tests in the study, indicating that GitHub Copilot helps developers write more functional code by a wide margin," the research highlighted.

The study also revealed that developers who utilised GitHub Copilot wrote more lines of code without encountering readability issues. Developers were able to write on average 13.6% more lines of code without readability errors when using GitHub Copilot. According to the research findings, "Improved readability: in blind reviews, code written with GitHub Copilot had significantly fewer code readability errors, allowing developers to write 13.6% more lines of code, on average, without encountering readability problems."

In addition to functionality and readability, the study measured other aspects of code quality, such as reliability, maintainability, and conciseness. Overall, the improved quality was statistically significant with readability up by 3.62%, reliability by 2.94%, maintainability by 2.47%, and conciseness by 4.16%. All the numbers were supported statistically. "Overall better quality code: readability improved by 3.62%, reliability by 2.94%, maintainability by 2.47%, and conciseness by 4.16%. All numbers were statistically significant," the study stated.

The study further showed that code written with GitHub Copilot was 5% more likely to be approved, emphasising the readiness of the code for integration and further development. "Higher approval rates: developers were 5% more likely to approve code written with GitHub Copilot, meaning that such code is ready to be merged sooner, speeding up the time to fix bugs or deploy new features," the findings underscored.

In the methodology of the study, participants were divided into two groups, with and without access to GitHub Copilot, to write API endpoints as a coding task. The submissions were scrutinised through unit tests and expert reviews from fellow developers, who analysed the code based on specific criteria such as readability, reliability, and maintainability without knowing which code had been written with the tool.

The research team posited that this study highlights GitHub Copilot's ability to improve code quality, helping developers focus more on refining the intricacies of their code. The results bolster previous findings that suggest developers feel more confident and maintain a better workflow when using GitHub Copilot.

The study makes a significant contribution as the first controlled study to explore GitHub Copilot's effect on code quality, suggesting its potential to enhance software development productivity and quality.