I couldn’t help but recall Microsoft’s Visual Studio IntelliCode. So, I had to try it to see if it lived up to the hype or was just another product destined to be forgotten.
At Baufest, we began our evaluation process by focusing on four key features we deemed most relevant for this AI tool: autocomplete, chat, unit test generation, and code exploration. We assessed its impact on a dozen projects in collaboration with our clients from various industries.
Autocomplete
Autocomplete is the “core” feature with which GitHub Copilot was launched. It offers contextual code suggestions, which, in our experience, are correct about 50% of the time. When suggestions don’t apply, they can simply be ignored, allowing developers to continue writing code traditionally. This process provides more context to the tool, increasing the likelihood of better recommendations in subsequent suggestions.
Chat
The editor’s chat feature does much more than suggest code. It’s not just a chat window—it recognizes the code a developer has written, displays error messages, and is integrated into the IDE (Integrated Development Environment). Developers can receive detailed analysis and explanations of what code blocks should do and even get proposals for bug fixes, new functionalities, or code restructuring.
Unit Test Generation
The integration within the code editor also includes the ability to select portions of code and request corresponding unit tests. When the code is well-structured, the unit test generation works very well. However, it struggles to produce useful code for complex scenarios or when many libraries with specific testing strategies are involved.
Code Exploration
The code exploration feature allows developers to select specific code sections and receive a detailed analysis of their structure and functionality. This tool is particularly useful for quickly understanding code written by others or in projects where prior experience is lacking. While code exploration is effective in providing summaries and identifying potential issues in well-organized code, it is less helpful for highly complex code or when multiple external dependencies require deep contextual understanding for accurate assessment.
Results
To understand Copilot’s impact, we used two approaches: a qualitative one, involving surveys, and a quantitative one, focusing on metrics for estimated versus actual time to complete tasks.
Here are the most interesting qualitative results:
- 97% of developers who used it recommend it and want to keep using it.
- 86% of developers believe it helped improve the speed of their development.
- 70% consider it a fundamental tool in their daily work.
“It’s a great tool for improving programming speed and aiding in code autocomplete, as well as generating or refactoring functions.”
For the quantitative results, we faced the undeniable challenge of measuring the impact on developer productivity, given that this activity is highly complex, collaborative, and largely creative work. To address this, we selected constrained technical tasks that were relatively easier to estimate and compared the actual time taken for each task. While these findings are anecdotal and specific to individual development teams (not generalizable to other teams), they reveal significant impacts worth noting.
- A team working on the evolutionary maintenance of core supermarket systems achieved a 25% average improvement in task times.
- A team maintaining integrations for a home improvement chain found similarly positive results, classified into four categories, as shown in the graph below.
- Another development team highlighted the impact on their code review tasks.
Conclusions
GitHub Copilot has transformed the way we approach development at Baufest. What began as a tool sparking curiosity has become a powerful assistant that complements our workflow. We’ve learned to trust its ability to accelerate tasks while understanding its limitations.
A practical tip: Don’t see Copilot as a replacement but as a teammate. Learn to interact with it, review its suggestions, and use them as a foundation to enhance and optimize your code.
Looking ahead, we are excited about how Copilot and similar tools will continue evolving, enabling us to focus more on creativity and solving complex problems while AI handles mundane tasks.
By Jesus Angulo, Account Architect at Baufest