GitHub diffs often fail

[ehuss]

I regularly see triagebot fail to fetch the diff of a PR, particularly when it is first opened. The error looks something like this:

ERROR request{uuid=b44d80a0-20ca-48af-992f-49aa869bf62e}: triagebot: handling event failed: failed to fetch diff comparison for https://api.github.com/repos/rust-lang/rust/compare/c8f94230282a8e8c1148f3e657f0199aad909228...b70763e306992af3e2b2cc06b7abdd8dfb7f2d88
Caused by:
    0: response: {"message":"Not Found","documentation_url":"https://docs.github.com/rest/commits/commits#compare-two-commits","status":"404"}
    1: HTTP status client error (404 Not Found) for url (https://api.github.com/repos/rust-lang/rust/compare/c8f94230282a8e8c1148f3e657f0199aad909228...b70763e306992af3e2b2cc06b7abdd8dfb7f2d88)

For some reason, GitHub is temporarily returning 404 for the /compare/ endpoint. I'm suspecting some kind of race condition on their backend.

This prevents triagebot from processing requests, particularly for assigning reviewers.

I do not know how to address this. One thought is to retry, maybe after a delay, but I don't think we can do that in the webhook window, which means triagebot would need to separately queue and process events. That seems complicated.

[xizheyin]

Is it possible to set up a retry mechanism, such as retrying 3 times, each time with an interval of a few tens of seconds?