I wanted to test this with a simple, reproducible scenario using CodeGuard AI, my Java 21 and Spring Boot-based GitHub pull request reviewer.

The experiment

I deliberately added an incorrect JUnit assertion to a GitHub pull request.

@Test
void shouldDetectIncorrectExpectedValue() {
    assertEquals(5, 2 + 2);
}

The assertion expects 5, but the expression evaluates to 4.

This is an intentionally failing test—not a hypothetical bug description.

What CodeGuard AI detected

CodeGuard reported:

• Test execution: FAILED
• Tests executed: 1
• Tests failed: 1
• Expected value: 5
• Actual value: 4

It also identified the failing assertion in the changed test file and suggested changing the expected value to 4.

That gave me concrete evidence that the reviewer could detect and explain this particular test failure.

What about the merge decision?

The interesting part was that CodeGuard recommended not merging, while its configured merge policy returned Merge After Fixes / WARN.

That difference matters.

Detecting a failure is one responsibility. Applying an appropriate merge policy is another. I want to validate both rather than assume that a correct finding automatically leads to the right merge decision.

What's next?

I'm testing CodeGuard AI across 100 scenarios covering:

• Genuine bugs and false positives
• Test execution and result verification
• Evidence quality and confidence
• Merge policies
• Suggested fixes and regression validation
• Repository context and reliability

I'll share the results—including failures and limitations—as testing continues.

Try CodeGuard AI

CodeGuard AI is built with Java 21 and Spring Boot, with support for AI providers including OpenAI, Gemini, and Ollama.

The source code is available on Gumroad:

https://javacoder716.gumroad.com/l/codeguard-ai?utm_source=hashnode&utm_medium=article&utm_campaign=codeguard_ai_testing&utm_content=test_09

I'd like to hear from other developers:

Should an AI code reviewer automatically block a pull request when its test suite fails, or should that decision remain configurable?

AI-assisted code review has made it much easier to scan a pull request for potential problems.

A model can identify security concerns, bugs, performance issues, code smells, and possible edge cases in seconds.

But there is a problem.

A developer doesn't necessarily need 20 findings.

They need to know:

• Which finding is actually important?

• Which one could affect production?

• How confident should we be in the finding?

• What part of the change needs human attention?

• Which findings can safely be ignored?

That distinction becomes increasingly important as AI-generated code becomes a larger part of software development.

More findings don't automatically mean better code review.

Better prioritization does.

Why severity alone isn't enough

A common approach is to assign findings a severity such as low, medium, high, or critical.

That's useful, but severity by itself doesn't tell the whole story.

Consider two findings:

Finding A

A possible null-pointer issue in code that is rarely executed.

Finding B

A small configuration change affecting a shared authentication component.

The first might look technically serious in isolation.

The second might have a much larger real-world impact.

This is why an AI review system needs more context than simply asking:

“Is this code wrong?”

It needs to help answer:

“How much attention should a developer give this change?”

That is a much more useful question for an engineering team.

Repository and pull-request context matters

A pull request should not always be treated as an isolated block of changed code.

A small change can have a large blast radius if it touches:

• shared services

• authentication or authorization

• database access

• configuration

• common libraries

• APIs used by multiple components

• infrastructure or deployment logic

This is where repository context becomes important.

The more context a review system can understand, the better it can distinguish between a harmless local change and a change that deserves deeper human investigation.

Confidence is different from severity

Another useful signal is confidence.

A high-severity finding with low confidence should not necessarily receive the same treatment as a high-severity finding with strong evidence.

For example:

High severity + high confidence

Investigate immediately.

High severity + low confidence

Worth investigating, but the reviewer should know that the model has uncertainty.

This distinction helps prevent AI review from becoming a source of unnecessary noise.

The goal isn't to make the AI sound certain.

The goal is to make the review more useful.

The human reviewer still matters

AI code review should not try to eliminate the developer from the process.

A better model is:

AI analysis → prioritization → human judgment → decision

The AI can perform the repetitive analysis.

The developer decides what actually matters in the context of the system, business requirements, architecture, and production environment.

This makes the AI reviewer an engineering assistant rather than an automatic authority.

What a useful AI code-review workflow should look like

A practical AI code-review workflow should focus on reducing the amount of unnecessary attention required from developers.

A useful workflow can look something like this:

1. Understand the pull request

Look at the actual changes, affected files, and surrounding repository context.

2. Analyze potential issues

Look for security, bugs, performance problems, maintainability concerns, and other engineering risks.

3. Structure the findings

Instead of returning a long list of unorganized observations, group the findings into meaningful categories.

4. Prioritize

Use signals such as severity, confidence, and potential impact to help the developer understand what deserves attention first.

5. Keep the developer in control

The final decision should remain with the engineering team.

This approach makes AI review less about generating the maximum number of findings and more about generating findings that are actually useful.

How we approached this with CodeGuard

These ideas were also the motivation behind building CodeGuard AI.

CodeGuard is designed as a self-hosted AI code-review application that integrates with GitHub pull requests.

Instead of treating AI output as the final answer, the workflow is designed around structured findings that can help developers understand:

• What was detected

• How severe the issue may be

• How confident the analysis is

• What part of the code is affected

• Which findings deserve closer human attention

CodeGuard also supports multiple AI providers, including OpenAI, Gemini, and Ollama.

Because the application is self-hosted source code, developers can inspect and customize the workflow instead of depending entirely on a hosted black-box service.

The bigger shift in AI-assisted development

As developers generate more code with AI, code review itself is going to change.

The question won't simply be:

“Can AI review this pull request?”

It will increasingly become:

“Can AI help the developer spend their limited review time on the changes that matter most?”

That distinction is important.

A system that produces more findings is not necessarily a better review system.

A system that helps a developer identify the findings worth investigating may be much more valuable.

AI should reduce review noise, not create another source of it.

Want to build and own this workflow?

CodeGuard AI is available as self-hosted source code for developers who want to build, inspect, and customize their own AI-powered GitHub code-review workflow.

It is built with Java 17 and Spring Boot and supports GitHub pull-request integration, structured findings, severity and confidence signals, and multiple AI providers.

Explore CodeGuard AI →