Skip to content
Open
Show file tree
Hide file tree
Changes from 2 commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -151,7 +151,9 @@ We employ several techniques to output a lot of value in a unit of time:
While some software houses lose \~30% of their performance (due to miscommunications, etc.), we stay way below 1% (the exact number is hard to measure when the amount of waste is so low).
As mentioned above, we tend to pause work on a ticket when we are not sure, choosing a small delay on a particular task over potentially having to discard work due to a bad assumption. Our clients prefer it this way.
10. Use modern IDEs. Pycharm, Cursor, Windsurf or VisualStudio with a LLM plugin. Some people are trying to use VIM with plugins for LLMs, but nowadays it's mostly Cursor/Windsurf/Pycharm.
11. Use LLMs (ChatGPT, GitHub Copilot, Claude etc) to speed up the work on the code, though watch every single byte of the diff like it's been written by a party you shouldn't trust.
11. Use LLMs to speed up coding, but we still own QA: read every byte and aim for our normal quality. For small,
contained “one‑shot” changes we may relax QA—and even skip review—if we save the prompt/spec so future bugs can be
fixed smoothly. See the [LLM‑assisted coding agreement](agreements.md#llm-assisted-coding).

# Code Review

Expand Down
50 changes: 47 additions & 3 deletions agreements.md
Original file line number Diff line number Diff line change
Expand Up @@ -166,6 +166,47 @@ That's one of the reasons we have someone in that role for every project.
This also allows for other solutions such as, for example, splitting the cost of a fix between two clients, discounts etc - something you couldn't do on your own.
Fortunately, with a dedicated client contact person, you don't have to!

## LLM-Assisted Coding

Motivation

- LLMs let us move faster (including during reviews), but cumulative low-quality changes can erode long-term
maintainability — one of our trademarks.
- Responsibility remains human: both the author and the reviewer own the quality of what ships, regardless of tooling.
- We may intentionally accept slightly lower quality for one‑shot, low‑risk changes to gain speed, but we do so when it is
clearly safe and we document the prompt/spec.
- There’s a trade‑off: more features with less quality vs fewer features with higher quality. Over time, lower quality
makes teams ship fewer features due to maintenance drag; we optimize for sustained maintainability while using
one‑shots to keep speed where risk is minimal.
- Practical observation: there’s a limit to how much an LLM can safely rewrite in a day without agreed QA.

Scope and rules

- Default path: normal QA and review. One‑shot changes (small, independently judgeable changes) may be merged with
relaxed quality and may skip review if all of the following hold:
- The change is low‑risk, contained, and has minimal blast radius; the author performs a basic functional check.
- The final “prompt/spec” is saved in the repository as markdown and kept up to date when the code changes.
- The conversation history may be condensed into a short “final prompt/summary” rather than stored verbatim.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I don't quite understand this guide. What does it mean "final" promt, does it mean I should save only the last promt I input into LLM? What is a purpose of this?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

relaxed the thing, I hope you will like it :)

- PR/commit mentions that the change is LLM‑assisted and links to the prompt/spec.
- Larger/core contributions done with LLM assistance require a case‑by‑case, agreed QA/release plan. If we must merge a
larger LLM change before full QA, do it consciously with that plan in place. Do not merge low‑quality core code
without such agreement.
- Use case‑by‑case judgement for one‑shots; if uncertain, take the normal QA/review path.

Author responsibilities

- Read every byte of generated code, functionally test it, and prepare the change for regular review (structure the diff,
write tests/docs where applicable).
- Include a link to the prompt/spec in the PR or commit description. Do not include secrets or client‑sensitive data in
prompts.

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

It is unclear to me - should I link to the prompt every time I use llm in my PR even if I review and refactor it? Why?

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

good catch, no, it is not needed, I relaxed the requirement of saving the prompt


Review expectations

- Normal review applies; the reviewer may use or not use LLM tooling.
- Verify that the prompt/spec exists, is up to date, and meaningfully reflects the change.
- Confirm maintainability isn’t degraded (structure, naming, tests, docs), and that no secrets are stored in
prompts/specs.

## Fast track decisions via Slack instead of standard Sociocracy approach

At Reef Technologies, we mostly make decisions in our weekly Sociocracy meetings.
Expand Down Expand Up @@ -240,11 +281,14 @@ See the new [Equipment Funding Policy](https://github.com/reef-technologies/hand

## LLM tool use & funding

Each Reef Technologies senior staff member is eligible to receive company-funded LLM service subscription (e.g. GitHub Copilot, ChatGPT Plus).
The subscription is paid by Reef Technologies for entire time it is being used it for any tasks related to Reef Technologies projects.
Each Reef Technologies senior staff member is eligible to receive company-funded LLM service subscription (e.g. GitHub
Comment thread
mateuszsrebrny-reef marked this conversation as resolved.
Outdated
Copilot, ChatGPT Plus).
The subscription is paid by Reef Technologies for entire time it is being used it for any tasks related to Reef
Comment thread
mateuszsrebrny-reef marked this conversation as resolved.
Outdated
Technologies projects.
Please make sure the subscription is terminated if you don't need it anymore to prevent license waste.

Please note, while our default contract with clients allows use of LLMs, make sure to check with a project manager before using LLM on any particular project.
Please note, while our default contract with clients allows use of LLMs, make sure to check with a project manager
before using LLM on any particular project.
As of mid 2023 every client has approved LLMs except one that is still considering it.
As for work done for Reef Technologies internal projects, you are free to use LLMs as you see fit.
In all cases you are responsible for quality of your work, regardless if LLM was used to produce it or not.
Expand Down
12 changes: 12 additions & 0 deletions docs/Code_Review.md
Original file line number Diff line number Diff line change
Expand Up @@ -12,6 +12,11 @@ As with every policy, if something is bad and you care about it enough, feel fre
and the governance process, operating in front of you and with you, will decide whether to change (most likely yes).
</details>

Responsibility remains with humans: authors and reviewers are accountable for code quality whether or not LLMs were
used. See the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding).
Tiny, low‑risk one‑shot LLM changes may skip review if they meet the agreement’s conditions (prompt/spec saved, low
blast radius, basic functional check). Otherwise, request review as usual.

## When/how to request a code review

1. If you are a developer, always perform basic functional testing (manually!) of your code and self-review your PR
Expand All @@ -31,6 +36,11 @@ and the governance process, operating in front of you and with you, will decide
commits instead, like “Fix XYZ”, “Add tests”. <details><summary>Note.</summary>This is a limitation of GitHub and a
primary motivator to consider moving development off to gitlab / gerrit, though as of writing this document we have
not decided to switch.</details>
6. Follow the [LLM‑Assisted Coding agreement](../agreements.md#llm-assisted-coding) and link to the saved prompt/spec.
If you’re using the LLM one‑shot relaxation, say so explicitly; those relaxed changes should stand alone, not be
Comment thread
mateuszsrebrny-reef marked this conversation as resolved.
Outdated
tucked into larger PRs. Authors must read and functionally verify all generated code before requesting review; if
merging without review under the one‑shot rule, mention “llm‑one‑shotted” and the prompt/spec path in the commit
message instead.

## How to review

Expand Down Expand Up @@ -101,6 +111,8 @@ Note: not all of these apply to all PRs, of course.
12. Performance / memory considerations. Will it OOM in a corner case after the change?
13. Self-healing. If a (network, HDD) device or a part of the system breaks temporarily, will the rest of the system
recover automatically or will it require manual intervention?
14. For LLM‑assisted changes: is the prompt/spec saved, linked in the PR, and kept up to date?
15. Maintainability not degraded (structure, naming, tests, docs); no secrets in prompt/specs.

## How to resolve the reviewer’s comments

Expand Down