|
| 1 | +# Contributing |
| 2 | + |
| 3 | +Guide for contributing to the Braintrust Python SDK. |
| 4 | + |
| 5 | +## Repository Structure |
| 6 | + |
| 7 | +``` |
| 8 | +braintrust-sdk-python/ |
| 9 | +├── py/ # Python SDK |
| 10 | +│ ├── src/braintrust/ # Source code |
| 11 | +│ │ ├── wrappers/ # Provider wrappers (OpenAI, Anthropic, Google, etc.) |
| 12 | +│ │ ├── contrib/ # Community integrations (Temporal, etc.) |
| 13 | +│ │ ├── devserver/ # Local dev server / CLI |
| 14 | +│ │ └── conftest.py # Shared test fixtures |
| 15 | +│ ├── noxfile.py # Test session definitions |
| 16 | +│ └── Makefile # Build/test commands |
| 17 | +├── integrations/ |
| 18 | +│ ├── langchain-py/ # LangChain integration |
| 19 | +│ └── adk-py/ # Google ADK integration |
| 20 | +├── internal/ # Golden tests |
| 21 | +├── scripts/ # Dev scripts |
| 22 | +├── docs/ # Documentation |
| 23 | +└── Makefile # Top-level commands |
| 24 | +``` |
| 25 | + |
| 26 | +## Setup |
| 27 | + |
| 28 | +### Prerequisites |
| 29 | + |
| 30 | +- Python 3.10+ (3.9 supported but some test sessions are skipped) |
| 31 | +- [uv](https://github.com/astral-sh/uv) (installed automatically by `make install-dev`) |
| 32 | + |
| 33 | +### Getting Started |
| 34 | + |
| 35 | +```bash |
| 36 | +# Clone the repo |
| 37 | +git clone https://github.com/braintrustdata/braintrust-sdk-python.git |
| 38 | +cd braintrust-sdk-python |
| 39 | + |
| 40 | +# Create venv and install all dependencies |
| 41 | +make develop |
| 42 | + |
| 43 | +# Activate the environment |
| 44 | +source env.sh |
| 45 | +``` |
| 46 | + |
| 47 | +### Python SDK Development |
| 48 | + |
| 49 | +```bash |
| 50 | +cd py |
| 51 | + |
| 52 | +# Install dev dependencies |
| 53 | +make install-dev |
| 54 | + |
| 55 | +# Install optional provider packages (for wrapper development) |
| 56 | +make install-optional |
| 57 | +``` |
| 58 | + |
| 59 | +## Running Tests |
| 60 | + |
| 61 | +### Python SDK |
| 62 | + |
| 63 | +Tests use [nox](https://nox.thea.codes/) to run across different dependency versions. |
| 64 | + |
| 65 | +```bash |
| 66 | +cd py |
| 67 | + |
| 68 | +# Run all test sessions |
| 69 | +make test |
| 70 | + |
| 71 | +# Run core tests only (no optional dependencies) |
| 72 | +make test-core |
| 73 | + |
| 74 | +# List all available sessions |
| 75 | +nox -l |
| 76 | + |
| 77 | +# Run a specific session |
| 78 | +nox -s "test_openai(latest)" |
| 79 | +nox -s "test_anthropic(latest)" |
| 80 | +nox -s "test_temporal(latest)" |
| 81 | + |
| 82 | +# Run a single test within a session |
| 83 | +nox -s "test_openai(latest)" -- -k "test_chat_metrics" |
| 84 | +``` |
| 85 | + |
| 86 | +### Integration Tests |
| 87 | + |
| 88 | +```bash |
| 89 | +# LangChain |
| 90 | +cd integrations/langchain-py |
| 91 | +uv sync |
| 92 | +uv run pytest src |
| 93 | + |
| 94 | +# ADK |
| 95 | +cd integrations/adk-py |
| 96 | +uv sync |
| 97 | +uv run pytest |
| 98 | +``` |
| 99 | + |
| 100 | +### Linting |
| 101 | + |
| 102 | +```bash |
| 103 | +# From repo root — runs pre-commit hooks (formatting, etc.) |
| 104 | +make fixup |
| 105 | + |
| 106 | +# Python-specific lint (pylint) |
| 107 | +cd py && make lint |
| 108 | +``` |
| 109 | + |
| 110 | +## VCR Cassette Testing |
| 111 | + |
| 112 | +Tests for API provider wrappers use VCR.py to record and replay HTTP interactions. This means most tests run without real API keys. |
| 113 | + |
| 114 | +See [docs/vcr-testing.md](docs/vcr-testing.md) for full details. Key points: |
| 115 | + |
| 116 | +- **Locally:** VCR records new cassettes on first run (`record_mode="once"`). You need a real API key to record. |
| 117 | +- **In CI:** VCR only replays existing cassettes (`record_mode="none"`). No API keys needed. |
| 118 | +- **Modifying tests:** If your change alters the HTTP request a test makes, you must re-record the cassette locally with a real API key and commit it. |
| 119 | +- **New tests:** Add `@pytest.mark.vcr`, record the cassette locally, and commit the cassette file. |
| 120 | + |
| 121 | +## CI Overview |
| 122 | + |
| 123 | +CI runs on GitHub Actions. All workflows are in `.github/workflows/`. |
| 124 | + |
| 125 | +### Workflows |
| 126 | + |
| 127 | +| Workflow | File | Trigger | What it does | |
| 128 | +|---|---|---|---| |
| 129 | +| **py** | `py.yaml` | PR (py/integrations changes), push to main | Runs nox test matrix across Python 3.10–3.13 on Ubuntu + Windows, plus integration tests | |
| 130 | +| **langchain-py** | `langchain-py-test.yaml` | Called by `py.yaml` | Lint + tests for the LangChain integration | |
| 131 | +| **adk-py** | `adk-py-test.yaml` | Called by `py.yaml` | Lint + tests for the Google ADK integration | |
| 132 | +| **lint** | `lint.yaml` | PR | Pre-commit hooks and formatting checks | |
| 133 | +| **publish** | `publish-py-sdk.yaml` | Tag push (`py-sdk-v*.*.*`) | Build, test wheel, publish to PyPI, create GitHub release | |
| 134 | +| **test-publish** | `test-publish-py-sdk.yaml` | Manual dispatch | Publish to TestPyPI for pre-release validation | |
| 135 | + |
| 136 | +### No API Key Secrets Required |
| 137 | + |
| 138 | +CI workflows do **not** use real API key secrets (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `GEMINI_API_KEY`). Tests rely on VCR cassettes with dummy API keys provided by test fixtures. This means: |
| 139 | + |
| 140 | +- Forks can run CI without configuring any secrets. |
| 141 | +- The `test_latest_wrappers_novcr` nox session (which disables VCR) is automatically skipped in CI. |
| 142 | + |
| 143 | +### Test Sharding |
| 144 | + |
| 145 | +The main `py.yaml` workflow shards nox sessions across 2 parallel jobs per Python version/OS combination using `scripts/nox-matrix.sh`. |
| 146 | + |
| 147 | +## Test Fixtures |
| 148 | + |
| 149 | +Key auto-applied fixtures defined in `py/src/braintrust/conftest.py`: |
| 150 | + |
| 151 | +| Fixture | Purpose | |
| 152 | +|---|---| |
| 153 | +| `setup_braintrust` | Sets dummy API keys (OpenAI, Google, Anthropic) for VCR tests | |
| 154 | +| `override_app_url_for_tests` | Points `BRAINTRUST_APP_URL` to production for consistent behavior | |
| 155 | +| `reset_braintrust_state` | Resets global SDK state after each test | |
| 156 | +| `skip_vcr_tests_in_wheel_mode` | Skips VCR tests when testing from an installed wheel | |
| 157 | + |
| 158 | +The `memory_logger` fixture (from `braintrust.test_helpers`) lets you capture logged spans in-memory without a real Braintrust connection: |
| 159 | + |
| 160 | +```python |
| 161 | +def test_something(memory_logger): |
| 162 | + # ... exercise code that logs spans ... |
| 163 | + spans = memory_logger.pop() |
| 164 | + assert len(spans) == 1 |
| 165 | +``` |
| 166 | + |
| 167 | +## Submitting Changes |
| 168 | + |
| 169 | +1. Create a branch for your changes. |
| 170 | +2. Make your changes and add/update tests. |
| 171 | +3. If you modified VCR tests, re-record cassettes and commit them. |
| 172 | +4. Run `make fixup` to format and lint. |
| 173 | +5. Run relevant test sessions to verify (e.g. `nox -s "test_openai(latest)"`). |
| 174 | +6. Open a pull request against `main`. |
0 commit comments