Should an AI code reviewer publicly label pull requests as low-effort AI output?
Our AI review bot can now tag pull requests that look like careless AI output: huge diffs for tiny tickets, duplicated helpers, tests that only check mocks. It's mostly right, but it once tagged a hand-written PR from a junior developer in front of the whole team. Would you turn on a label like that, and who should be able to see it?
The signals themselves are useful: a diff far bigger than the ticket, new helpers that duplicate existing ones, and tests that only exercise mocks are worth flagging whoever wrote the code. The problem is the word and the audience, not the check.
A safer setup is to show the finding to the author first, phrased as specific issues rather than a verdict, and only make it visible to the team after a person has looked at it. A false positive then costs one private comment instead of someone's reputation.
The bigger win is moving the check earlier. If the author, or their agent, runs a review pass before opening the pull request, most of these problems never reach the team at all. Ask for a separate reviewer pass against the ticket, and require test output as evidence before anything is called done.
Keep track of how often the label is wrong. If it is regularly misfiring on hand-written code, tune the rules or drop the label and keep only the line-level comments.
Listings mentioned
- Requesting code review · skill by obraSends finished work to a separate reviewer before merging, so problems are caught before teammates see them.
- PR review expert · skill by alirezarezvaniReviews a pull request diff for quality and security issues with specific, line-level findings.
- Code Review Reception · skill by obraHelps the author check review feedback on its merits instead of accepting or rejecting it blindly.
Answers by the AgentAlley team, drafted with AI and checked against the listings they link to. Not a real-person reply from the original thread.