Playing with GitHub Copilot Code Review

Promising, but I have mixed feelings

Photo by Volodymyr Dobrovolskyy on Unsplash

Code Review is one of the async agentic capabilities that github.com now offers. It can run in some IDEs (Visual Studio and VS Code), but the best use case in my opinion is when it runs automatically and asynchronously in your repo in github.com.

To do that, you need to enable those settings in the repository settings, with rulesets. You can create a default one that triggers Code Review after every PR, but you can specify a more complex rule set if necessary. It’s also recommended to create custom instructions, which are basically a system prompt that is given to Code Review’s context when it executes.

In my case, I created some simple SQL style guide instructions in a markdown file (.github/instructions/sql.instructions.md) that apply to any SQL file anywhere in the repo

These are rather simple and standard SQL style guide instructions that can be implemented by a linter, but I was curious to see Code Review in actions.

I created a simple file that is supposed to be a task in an Airflow DAG, which violates some of the style guide instructions.

As soon as I create the pull request, Code Review kicks in automatically ...

… suggesting these four changes:

The first two refers to the usage of lower case in SQL statements, the third one is about the missing alias, and the fourth is about the preferred style for WHERE clauses. It’s interesting how Code Review confirms that it’s using the repo custom instructions, and it offers to implement the suggestion.

Note that there is no suggestion about:

• Replacing the trailing commas in the SELECT statement with leading commas

• Adding a new line after the FROM statement

• Using 2 spaces for indentation

I tell Code Review to implement suggestion 1, 2, and 4, which triggers 3 PRs.

The first one implements all the changes, including the leading commas in the SELECT statement and the 2 spaces for indentation, although these style guide violations were not flagged by Code Review.

The second PR fails with no explanation in the detailed logs 🤷‍♂️.

The third PR fixes the lower case and the WHERE clause, but not the trailing commas in the SELECT statement, which remains untouched and thus keep also the 4 spaces for indentation.

This was a rather simple test for Code Review, mainly related to style that could be enforced by a good ol’ linter. No bug fixing or something more complicated. Code Review did ok-ish, but in a very convoluted way, with 3 open PRs of which only one was more or less correct (it didn’t catch all the violations). Being able to express instructions in natural language is great compared to something like this:

However, it’s a bit concerning that it’s not able to be at par with a linter. If Code Review cannot deterministically intercept a predefined set of “bugs”, it’s unclear that at this stage it would be able to perform more complex bug fixing.

---

Do you like this post? Of course you do. Share it on Twitter/X, LinkedIn and HackerNews