Code-review agent

Design an AI agent that reviews every pull request across 2,000 repositories.

45:00

45 minutes in five phases. Talk out loud the whole time.

Ask first

Which languages, and how big are the repositories?

What should it catch: bugs, security issues, style?

Where do its comments go?

How many false alarms will developers tolerate?

Can code leave the company network?

Trigger on each pull request through a webhook or CI job.

Gather context: the diff, related files from code search, and the repository’s conventions file.

Separate passes for correctness, security and tests.

De-duplicate and rank findings, and post only high-confidence inline comments.

Collect feedback (accepted or dismissed) to tune prompts and thresholds.

Evaluate on historical pull requests with known bugs.

Full-repository context versus cost.

Precision versus recall. Noise kills adoption.

Static analysis plus an LLM versus an LLM alone.

Track the acceptance rate for each type of comment and cut the low-value ones.

Use enterprise endpoints that don’t train on your data inside a private network, or a self-hosted model.

Treat all pull request content as untrusted. The agent can only comment; it can never merge or run code.

Pull requests per day × tokens per review × price.

A target comment acceptance rate, such as above 50%.