Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Batch processing #32

Open
wants to merge 30 commits into
base: main
Choose a base branch
from
Open

Batch processing #32

wants to merge 30 commits into from

Conversation

berkeli
Copy link
Owner

@berkeli berkeli commented Nov 7, 2022 •
edited
Loading

Extensions:

  1. The tool should write an output CSV with the URLs that failed, in a format that could be used as input: go run . --input input.csv --output output.csv --output-failed failed.csv
  2. Failures in downloading and uploading the images can be temporary: introduce a retry-with-backoff strategy to make the tool more resilient to temporary failures
  3. Do not re-upload the same image to S3
    • Can you do this without storing anything in a database?
  4. Do not download & process the same image
    • Can you also do this without storing anything in a database?
  5. To speed up the tool, process and upload in parallel using goroutines

@berkeli berkeli added this to the Sprint 2 milestone Nov 7, 2022
@berkeli berkeli linked an issue Nov 7, 2022 that may be closed by this pull request
Batch Processing Project #23
Open
9 tasks
@berkeli berkeli changed the title Batch processing 3 Batch processing Nov 7, 2022
illicitonion
illicitonion reviewed Nov 21, 2022
View reviewed changes
Copy link

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

This is looking really well put together, a very polished! Nicely put together!

batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/helpers.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
illicitonion
illicitonion reviewed Nov 27, 2022
View reviewed changes
batch-processing/pipeline.go Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Show resolved Hide resolved
batch-processing/pipeline.go Show resolved Hide resolved
batch-processing/pipeline_test.go Outdated Show resolved Hide resolved
batch-processing/pipeline.go Outdated Show resolved Hide resolved
illicitonion
illicitonion reviewed Nov 27, 2022
View reviewed changes
Copy link

@illicitonion illicitonion left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Looks really nice - great refactoring!

batch-processing/pipeline_test.go Outdated Show resolved Hide resolved
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Reviewers

@illicitonion illicitonion illicitonion left review comments

Assignees
No one assigned
Labels
None yet
Projects
None yet
Milestone
Sprint 2
Development

Successfully merging this pull request may close these issues.

Batch Processing Project
2 participants
@berkeli @illicitonion