Add PCW validation in CharactersLogger tests #7879
Conversation
akalia25
changed the title
akalia25
changed the title
|
@hitesh-1997 / @valerybugakov, any chance I could get a review or suggestions on how I could make this PCW test better? 🙏 Thanks! |
| return { humanCode, codyCode } | ||
| } | ||
|
|
||
| function calculateHumanInsertions(flushEvent: any): number { |
There was a problem hiding this comment.
cross checking if flushEvent should have any type apart from any
| ): number { | ||
| // PCW = (Cody code written - Cody code removed) / Total code written | ||
| const codyNetContribution = codyInsertions - codyDeletions | ||
| const totalCodeWritten = humanInsertions + codyInsertions |
There was a problem hiding this comment.
Just cross checking if totalCodeWritten should be totalCodeWritten = humanNetContribution + codyNetContribution
where netContribution = insertions - deletions.
Not related to this PR but thinking out loud and cross checking if this makes sense in general:
do you think instead of percentage of code written, percent of code editable is a better metrics. for example, in case of code cleaning, deletions are good contribution and although netContribution could be negative b/c deletions are more than insertion but overall this highlights a better contribution if it makes to the final commit.
| } | ||
|
|
||
| // Calculate true PCW using all events | ||
| const PCW = totalCodyInsertions / (totalHumanInsertions + totalCodyInsertions) |
There was a problem hiding this comment.
Just cross checking since its insertion, we are not taking deletions into account no
This PR adds PCW (Percentage of Code Written by Cody) tests for
CharactersLogger. More info on PCW metric herehttps://sourcegraph.com/docs/analytics/pcwTest plan
Existing tests pass + all new test scenarios have been verified to pass with correct PCW calculations.