Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Retry the file if parser fails #71

Open
wants to merge 6 commits into
base: main
Choose a base branch
from
Open

Conversation

ankush-cohere
Copy link
Contributor

@ankush-cohere ankush-cohere commented Dec 20, 2024

Fixes

ChunkedEncodingError: ('Connection broken: IncompleteRead(4632016 bytes read, 10217602 more expected)', IncompleteRead(4632016 bytes read, 10217602 more expected))

This issue was caused by a parser restarting or dying because of some reason. The request to parser failed and it was not handled gracefully. This change will retry the file X times (3 by default) and will fail gracefully with an error log if not able to parse the file

Adding multiple retries to process_file

@ankush-cohere ankush-cohere requested a review from a team as a code owner December 20, 2024 12:08
@ankush-cohere ankush-cohere force-pushed the fix/bug_parser_dies branch 2 times, most recently from c420c6a to fdc732c Compare December 20, 2024 12:15
@ankush-cohere ankush-cohere marked this pull request as draft December 20, 2024 13:04
@ankush-cohere ankush-cohere marked this pull request as ready for review December 20, 2024 13:17
@ankush-cohere ankush-cohere marked this pull request as draft December 20, 2024 13:21
javier-cohere
javier-cohere previously approved these changes Dec 20, 2024
corafid
corafid previously approved these changes Dec 22, 2024
cohere/compass/utils.py Outdated Show resolved Hide resolved
luca-cohere
luca-cohere previously approved these changes Dec 23, 2024
@ankush-cohere ankush-cohere marked this pull request as ready for review January 6, 2025 08:39
@ankush-cohere ankush-cohere force-pushed the fix/bug_parser_dies branch 2 times, most recently from e79d8ae to bec120c Compare January 6, 2025 12:27
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

Successfully merging this pull request may close these issues.

4 participants