AI Generated Code

[ryneeverett]

Should @GothenburgBitFactory have a project-wide policy on AI generated code? We're only going to see more of this in the future and it might be advantageous to think about it in the abstract when we're not facing an immediate concrete decision. In particular, it seems we need to balance:

• respecting reviewer's time
• preserving code quality
• preserving open-source license integrity
• welcoming contributors with non-traditional programming backgrounds
• encouraging disclosure/attribution of generated code

[djmitche]

I agree! Do you know of examples of policies from other projects?

[djmitche]

I do like that formulation, and especially the concise description of the harms of AI, with references!

I think next steps would be

• Getting some feedback from other GBF members (@lauft, @tbabej, @linuxcaffe) as to whether to adopt an org-wide policy, and if so how to encode that
• Actually writing down the policy and ensuring it's linked from the relevant contribution guidelines, PR templates, etc.

[ryneeverett]

The actual context in which I came across it was when somebody made a highly technical handwritten report which incorporated a 1-line diff generated with AI and their speculations about why this diff fixed a bug. Then somebody linked the AI policy and they were driven away. I don't think we want that.

IMO postmarket's policy is unnecessarily political for a software community. We don't need to alienate AI optimists -- who no doubt believe that with enough training AI will solve the environment and usher in the post-scarcity society -- as long as their behavior toward our community is respectful.

[djmitche]

Huh, that situation doesn't seem to fit the linked policy, and certainly driving away people thinking hard about the application is not the desire.

[tbabej]

My 2 cents are that I think the risk of us getting in trouble due to AI generated code is relatively low, while the likelihood of us driving potentially useful contributions if we enact a strict AI policy is a much more likely scenario. Even in the unlikely case of us accepting proprietary code changes, it would not be that hard to simply remove them from codebase.

I am more concerned about people contributing low-quality, but good-looking PRs that only look good at a cursory glance because they have been mostly AI-generated without the (human) author/supervisor actually understanding the codebase. How to prevent that scenario, is unclear to me at this time.

[lauft]

To me the problem with AI generated code seems to be that it simplifies to create large PRs which then require time and effort to review. Especially if someone "just vibed it".

I agree on addressing the issue, e.g. in the CONTRIBUTING documents, maybe referencing a central statement of the GBF. I would not copy the postmarket policy, though...