Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

[bug]: Claims can be incorrectly identified as supported #6

Open
CalebCourier opened this issue Jan 7, 2025 · 0 comments
Open

[bug]: Claims can be incorrectly identified as supported #6

CalebCourier opened this issue Jan 7, 2025 · 0 comments

Comments

@CalebCourier
Copy link

When filtering out supported claims, this line assumes that concurrent.futures.as_completed() maintains order, which it does not, since it is zipping the claims as originally ordered with the future results as they are returned.

This means that a claim can be incorrectly identified as supported if subsequent calls made in _inference_remote return out of order (i.e. one request takes longer than another made after it).

See contrived code sample below for a simplified example:

import concurrent.futures
import time

def task(n):
    time.sleep(n)
    return n

claims = [3, 1, 2]

with concurrent.futures.ThreadPoolExecutor() as executor:
    futures = [executor.submit(task, claim) for claim in claims]
    is_supported = [future.result() for future in concurrent.futures.as_completed(futures)]


supported_claims = [
    (f"claim: {c}", f"is_supported: {s}")
    for c, s in zip(claims, is_supported)
]

print(supported_claims)

One solution would be to return a composite response from _inference_remote that includes both the original claim and the FactcheckCreateResponse for that claim.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

1 participant