Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch Processing Project #23

Open
9 tasks done
berkeli opened this issue Nov 7, 2022 · 0 comments · May be fixed by #32
Open
9 tasks done

Batch Processing Project #23

berkeli opened this issue Nov 7, 2022 · 0 comments · May be fixed by #32
Assignees
@berkeli
Labels
Large day or more priority-normal
Milestone
Sprint 2

Comments

@berkeli
Copy link
Owner

berkeli commented Nov 7, 2022 •
edited
Loading

https://github.com/CodeYourFuture/immersive-go-course/tree/main/batch-processing

  1. Read the requirements
  2. Implementation

Extensions:

  1. The tool should write an output CSV with the URLs that failed, in a format that could be used as input: go run . --input input.csv --output output.csv --output-failed failed.csv
  2. Failures in downloading and uploading the images can be temporary: introduce a retry-with-backoff strategy to make the tool more resilient to temporary failures
  3. Do not re-upload the same image to S3
    • Can you do this without storing anything in a database?
  4. Do not download & process the same image
    • Can you also do this without storing anything in a database?
  5. To speed up the tool, process and upload in parallel using goroutines
@berkeli berkeli self-assigned this Nov 7, 2022
@berkeli berkeli moved this from Todo to In Progress in Immersive Go Course Nov 7, 2022
@berkeli berkeli added this to the Sprint 2 milestone Nov 7, 2022
@berkeli berkeli moved this from In Progress to In Review in Immersive Go Course Nov 7, 2022
@berkeli berkeli linked a pull request Nov 7, 2022 that will close this issue
Batch processing #32
Open
7 tasks
@berkeli berkeli moved this from In Review to Done in Immersive Go Course Nov 25, 2022
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Assignees

@berkeli berkeli

Labels
Large day or more priority-normal
Projects
Immersive Go Course
Status: Done
Milestone
Sprint 2
Development

Successfully merging a pull request may close this issue.

Batch processing berkeli/immersive-go
1 participant
@berkeli