-
Notifications
You must be signed in to change notification settings - Fork 2
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Add new pipeline 2025-03-6 #2846
base: main
Are you sure you want to change the base?
Conversation
☂️ Python Coverage
Overall Coverage
New FilesNo new covered files... Modified FilesNo covered modified files...
|
b78d6d3
to
f1ada4b
Compare
Approved for reindex 👍 |
f1ada4b
to
9689608
Compare
9689608
to
0a4a359
Compare
@@ -52,7 +52,7 @@ module "id_minter_lambda" { | |||
queue_config = { | |||
topic_arns = local.transformer_output_topic_arns | |||
max_receive_count = 3 | |||
maximum_concurrency = 10 | |||
maximum_concurrency = 30 |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others. Learn more.
When the ID minter concurrency is set to 10, it becomes a bottleneck during a reindex. Increasing the concurrency to 30 helps, without overwhelming the database.
What does this change?
#5912
Following on wellcomecollection/platform#5911.
How to test
How can we measure success?
After reindexing with the new pipeline, all concepts which reference the same source concept or have the same labels (for label-derived concepts) should have the same canonical id.
Have we considered potential risks?
Risks should be minimal.