Skip to content

Commit e64f371

Browse files
authored
Review Agent (#298)
* push review agent implementation * feedback * wip integrating review agent into monorepo * move review agent to web * feedback * feedback * add rate limit throttling to octokit * configure agent ui in app * docs * add review command logic and add logging for review agent to data cache dir * fix bug with llm returning multiple reviews in single invocation * fix doc link bug * feedback and improved docs for review agent * review agent doc nits * mcp doc nit
1 parent f385ee2 commit e64f371

29 files changed

+1168
-52
lines changed

.env.development

Lines changed: 13 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -23,6 +23,8 @@ AUTH_URL="http://localhost:3000"
2323
# AUTH_GOOGLE_CLIENT_ID=""
2424
# AUTH_GOOGLE_CLIENT_SECRET=""
2525

26+
#DATA_CACHE_DIR="" # Path to the sourcebot cache dir (ex. ~/sourcebot/.sourcebot)
27+
2628
# Email
2729
# EMAIL_FROM_ADDRESS="" # The from address for transactional emails.
2830
# SMTP_CONNECTION_URL="" # The SMTP connection URL for transactional emails.
@@ -51,6 +53,16 @@ REDIS_URL="redis://localhost:6379"
5153
# STRIPE_WEBHOOK_SECRET: z.string().optional(),
5254
# STRIPE_ENABLE_TEST_CLOCKS=false
5355

56+
# Agents
57+
58+
# GITHUB_APP_ID=
59+
# GITHUB_APP_PRIVATE_KEY_PATH=
60+
# GITHUB_APP_WEBHOOK_SECRET=
61+
# OPENAI_API_KEY=
62+
REVIEW_AGENT_LOGGING_ENABLED=true
63+
REVIEW_AGENT_AUTO_REVIEW_ENABLED=false
64+
REVIEW_AGENT_REVIEW_COMMAND=review
65+
5466
# Misc
5567

5668
# Generated using:
@@ -78,4 +90,4 @@ SOURCEBOT_TELEMETRY_DISABLED=true # Disables telemetry collection
7890
# NODE_ENV=
7991
# SOURCEBOT_TENANCY_MODE=single
8092

81-
# NEXT_PUBLIC_SOURCEBOT_CLOUD_ENVIRONMENT=
93+
# NEXT_PUBLIC_SOURCEBOT_CLOUD_ENVIRONMENT=

docs/docs.json

Lines changed: 7 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -43,6 +43,13 @@
4343
}
4444
]
4545
},
46+
{
47+
"group": "Agents",
48+
"pages": [
49+
"docs/agents/overview",
50+
"docs/agents/review-agent"
51+
]
52+
},
4653
{
4754
"group": "More",
4855
"pages": [

docs/docs/agents/overview.mdx

Lines changed: 17 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,17 @@
1+
---
2+
title: "Agents Overview"
3+
sidebarTitle: "Overview"
4+
---
5+
6+
<Note>
7+
Have an idea for an agent that we haven't built? Submit a [feature request](https://github.com/sourcebot-dev/sourcebot/discussions/categories/feature-requests) on our GitHub
8+
</Note>
9+
10+
Agents are automations that leverage the code indexed on Sourcebot to perform a specific task. Once you've setup Sourcebot, check out the
11+
guides below to configure additional agents.
12+
13+
<CardGroup cols={2}>
14+
<Card horizontal title="Review Agent" icon="gear" href="/docs/agents/review-agent">
15+
An AI agent that reviews your PRs to identify issues
16+
</Card>
17+
</CardGroup>

docs/docs/agents/review-agent.mdx

Lines changed: 93 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,93 @@
1+
---
2+
title: AI Code Review Agent
3+
sidebarTitle: AI Code Review Agent
4+
---
5+
6+
<Note>
7+
This agent sends data to OpenAI (through an API key you supply) to perform code reviews. This data includes code from the PR being reviewed, as well as additional relevant context from your
8+
codebase that the agent may fetch to perform the review.
9+
</Note>
10+
11+
This agent provides codebase-aware reviews for your PRs. For each diff, this agent fetches relevant context from Sourcebot and feeds it into an LLM for a detailed review of your changes.
12+
13+
The AI Code Review Agent is open source and packaged in [Sourcebot](https://github.com/sourcebot-dev/sourcebot). To get started using this agent, [deploy Sourcebot](/self-hosting/overview)
14+
and then follow the configuration instructions below.
15+
16+
![AI Code Review Agent Example](/images/review_agent_example.png)
17+
18+
# Configure
19+
20+
This agent currently only supports reviewing GitHub PRs. You configure the agent by creating a GitHub app, installing it into your GitHub organization, and then giving your app info to Sourcebot.
21+
22+
Before you get started, make sure you have an OpenAPI account that you can create an OpenAPI key with.
23+
24+
<Steps>
25+
<Step title="Register a GitHub app">
26+
Follow the official GitHub guide for [registering a GitHub app](https://docs.github.com/en/apps/creating-github-apps/registering-a-github-app/registering-a-github-app)
27+
28+
- GitHub App name: You can make this whatever you want (ex. Sourcebot Review Agent)
29+
- Homepage URL: You can make this whatever you want (ex. https://www.sourcebot.dev/)
30+
- Webhook URL (**IMPORTANT**): You must set this to point to your Sourcebot deployment at /api/webhook (ex. https://sourcebot.aperture.com/api/webhook). Your Sourcebot deployment must be able to accept requests from GitHub
31+
(either github.com or your self-hosted enterprise server) for this to work. If you're running Sourcebot locally, you can [use smee](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#step-2-get-a-webhook-proxy-url) to [forward webhooks](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#step-6-start-your-server) to your local deployment.
32+
- Permissions
33+
- Pull requests: Read & Write
34+
- Issues: Read & Write
35+
- Contents: Read
36+
- Events:
37+
- Pull request
38+
- Issue comment
39+
</Step>
40+
<Step title="Install the GitHub app in your organization">
41+
Navigate to your new [GitHub app's page](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#navigate-to-your-app-settings) and press `Install`
42+
</Step>
43+
<Step title="Configure the environment variables in Sourcebot">
44+
Sourcebot requires the following environment variables to begin reviewing PRs through your new GitHub app:
45+
46+
- `GITHUB_APP_ID`: The client ID of your GitHub app. Can be found in your [app settings](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#navigate-to-your-app-settings)
47+
- `GITHUB_APP_WEBHOOK_SECRET`: A random webhook secret that you've set in your [app settings](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#navigate-to-your-app-settings). This can be anything (ex. `python -c "import secrets; print(secrets.token_hex(10))"` to generate a random secret)
48+
- `GITHUB_APP_PRIVATE_KEY_PATH`: The path to your app's private key. If you're running Sourcebot from a container, this is the path to this file from within your container
49+
(ex `/data/review-agent-key.pem`). You must copy the private key file into the directory you mount to Sourcebot (similar to the config file).
50+
51+
You can generate a private key file for your app in the [app settings](https://docs.github.com/en/apps/creating-github-apps/writing-code-for-a-github-app/quickstart#navigate-to-your-app-settings). You must copy this private key file into the
52+
directory that you mount to Sourcebot
53+
![GitHub App Private Key](/images/github_app_private_key.png)
54+
- `OPENAI_API_KEY`: Your OpenAI API key
55+
- `REVIEW_AGENT_AUTO_REVIEW_ENABLED` (default: `false`): If enabled, the review agent will automatically review any new or updated PR. If disabled, you must invoke it using the command defined by `REVIEW_AGENT_REVIEW_COMMAND`
56+
- `REVIEW_AGENT_REVIEW_COMMAND` (default: `review`): The command that invokes the review agent (ex. `/review`) when a user comments on the PR. Don't include the slash character in this value.
57+
58+
You can find an example docker compose file below.
59+
- This docker compose file is placed in `~/sourcebot_review_agent_workspace`, and I'm mounting that directory to Sourcebot
60+
- The config and the app private key files are placed in this directory
61+
- The paths to these files are given to Sourcebot relative to `/data` since that's the directory in Sourcebot that I'm mounting to
62+
63+
```yaml
64+
services:
65+
sourcebot:
66+
image: ghcr.io/sourcebot-dev/sourcebot:latest
67+
pull_policy: always
68+
container_name: sourcebot
69+
ports:
70+
- "3000:3000"
71+
volumes:
72+
- "/Users/michael/sourcebot_review_agent_workspace:/data"
73+
environment:
74+
CONFIG_PATH: "/data/config.json"
75+
GITHUB_APP_ID: "my-github-app-id"
76+
GITHUB_APP_WEBHOOK_SECRET: "my-github-app-webhook-secret"
77+
GITHUB_APP_PRIVATE_KEY_PATH: "/data/review-agent-key.pem"
78+
OPENAI_API_KEY: "sk-proj-my-open-api-key"
79+
```
80+
</Step>
81+
<Step title="Verify configuration">
82+
Navigate to the agents page by pressing `Agents` in the Sourcebot nav menu. If you've configured your environment variables correctly you'll see the following:
83+
84+
![Review Agent Configured](/images/review_agent_configured.png)
85+
</Step>
86+
</Steps>
87+
88+
# Using the agent
89+
90+
The review agent will not automatically review your PRs by default. To enable this feature, set the `REVIEW_AGENT_AUTO_REVIEW_ENABLED` environment variable to true.
91+
92+
You can invoke the review agent manually by commenting `/review` on the PR you'd like it to review. You can configure the command that triggers the agent by changing
93+
the `REVIEW_AGENT_REVIEW_COMMAND` environment variable.

docs/docs/more/mcp-server.mdx

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -7,7 +7,7 @@ sidebarTitle: Sourcebot MCP server
77
This feature is only available when [self-hosting](/self-hosting) with [authentication](/self-hosting/more/authentication) disabled.
88
</Note>
99

10-
The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is a open standard for providing context to LLMs. The [@sourcebot/mcp](https://www.npmjs.com/package/@sourcebot/mcp) package is a MCP server that enables LLMs to interface with your Sourcebot instance, enabling MCP clients like Cursor, Vscode, and others to have context over your entire codebase.
10+
The [Model Context Protocol](https://modelcontextprotocol.io/introduction) (MCP) is an open standard for providing context to LLMs. The [@sourcebot/mcp](https://www.npmjs.com/package/@sourcebot/mcp) package is a MCP server that enables LLMs to interface with your Sourcebot instance, enabling MCP clients like Cursor, Vscode, and others to have context over your entire codebase.
1111

1212
## Getting Started
1313

21.7 KB
Loading
70.2 KB
Loading

docs/images/review_agent_example.png

211 KB
Loading

package.json

Lines changed: 3 additions & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -1,7 +1,8 @@
11
{
22
"private": true,
33
"workspaces": [
4-
"packages/*"
4+
"packages/*",
5+
"packages/agents/*"
56
],
67
"scripts": {
78
"build": "cross-env SKIP_ENV_VALIDATION=1 yarn workspaces foreach -A run build",
@@ -11,6 +12,7 @@
1112
"dev:zoekt": "yarn with-env zoekt-webserver -index .sourcebot/index -rpc",
1213
"dev:backend": "yarn with-env yarn workspace @sourcebot/backend dev:watch",
1314
"dev:web": "yarn with-env yarn workspace @sourcebot/web dev",
15+
"dev:review-agent": "yarn with-env yarn workspace @sourcebot/review-agent dev",
1416
"watch:mcp": "yarn workspace @sourcebot/mcp build:watch",
1517
"watch:schemas": "yarn workspace @sourcebot/schemas watch",
1618
"dev:prisma:migrate:dev": "yarn with-env yarn workspace @sourcebot/db prisma:migrate:dev",

packages/backend/src/env.ts

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -27,6 +27,8 @@ export const env = createEnv({
2727
SOURCEBOT_INSTALL_ID: z.string().default("unknown"),
2828
NEXT_PUBLIC_SOURCEBOT_VERSION: z.string().default("unknown"),
2929

30+
DATA_CACHE_DIR: z.string(),
31+
3032
NEXT_PUBLIC_POSTHOG_PAPIK: z.string().optional(),
3133

3234
FALLBACK_GITHUB_CLOUD_TOKEN: z.string().optional(),

0 commit comments

Comments
 (0)