Skip to content

perf: download products and assets in parallel #1890

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom
Draft
wants to merge 15 commits into
base: develop
Choose a base branch
from
Draft

perf: download products and assets in parallel #1890

wants to merge 15 commits into from
+966 −968

Conversation

@anesson-cs
Copy link
Collaborator

@anesson-cs anesson-cs commented Oct 21, 2025 •
edited by sbrunato
Loading

Fixes #146

Perform multiple products downloads from download_all() core method in parallel to increase significantly efficiency.
If a product has assets, they are also downloaded in parallel. In that case and if there are downloads in parallel, the same executor is used to not disturb the parallelism.

A user can decide to use his own executor by passing executor as positional argument of download() and download_all() core methods and download() method from an EOProduct instance. Otherwise, an executor by default without specification of concurrent.futures.ThreadPoolExecutor class will be used instead.

If a provider has limitations about the maximum number of workers from an executor to use, it will take over the current value of the executor if needed.

@anesson-cs anesson-cs changed the title feat: download products and assets in parallel perf: download products and assets in parallel Oct 21, 2025
@github-actions
Copy link
Contributor

github-actions bot commented Oct 21, 2025 •
edited
Loading

Test Results

    4 files  ±0      4 suites  ±0   3m 46s ⏱️ -5s
  610 tests ±0    609 ✅ ±0  1 💤 ±0  0 ❌ ±0 
2 440 runs  ±0  2 434 ✅ ±0  6 💤 ±0  0 ❌ ±0 

Results for commit aefce77. ± Comparison against base commit 0b88c50.

♻️ This comment has been updated with latest results.

@eodag-bot
Copy link
Collaborator

eodag-bot commented Oct 21, 2025 •
edited
Loading

badge

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         246      10  95.93%   103-114, 628
config.py                                      454      32  92.95%   81-83, 92, 100, 104-106, 177, 188, 205-206, 735-737, 861-864, 908-909, 918-919, 953, 1033, 1054, 1062, 1092-1097, 1099
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    760      36  95.26%   342, 609, 640, 682-685, 723, 767, 801, 846-851, 875, 961, 1193, 1330, 1373, 1386, 1453, 1466-1467, 1543-1548, 1560-1563, 1894, 2140, 2203-2204, 2233-2234
api/search_result.py                           180      15  91.67%   113, 122, 129, 143, 187, 204, 301, 433, 438-439, 473, 487, 510-511, 517
api/product/__init__.py                         18       2  88.89%   60, 62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        215      18  91.63%   174, 243-244, 262-263, 371, 400, 407, 515, 535, 559-562, 571-574, 621
api/product/metadata_mapping.py                775      53  93.16%   125-127, 218-223, 244, 299-300, 388, 409, 461-462, 499, 520-523, 546, 558-559, 600, 623, 653-658, 717-722, 734, 742, 974, 1134, 1143-1147, 1164-1169, 1302, 1325, 1334, 1356, 1361, 1413, 1485, 1506, 1532, 1546, 1571, 1617, 1686, 1758
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     23       0  100.00%
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      18  89.60%   105-110, 145-150, 154, 192, 214, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101      10  90.10%   178-180, 228-229, 255-257, 287-288
plugins/apis/usgs.py                           182      25  86.26%   161, 267, 301, 343-345, 350, 378-379, 384, 414-421, 432-437, 459-465
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-113
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                63      25  60.32%   62-65, 72-75, 81, 85, 89, 100-116, 131-157
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        395      73  81.52%   269, 303, 314-320, 361-364, 391-395, 509-511, 515, 546-547, 553-557, 588, 653-661, 725-818, 829-834, 872, 898, 943-945, 997
plugins/download/base.py                       273      37  86.45%   133, 162, 309-310, 368-369, 411, 415-426, 440, 506, 521-525, 564-565, 590-607, 609-613, 663, 684, 706, 714
plugins/download/http.py                       551      77  86.03%   238, 280-283, 345-348, 351, 358-363, 394-396, 413, 428, 486, 521, 535, 549, 557-561, 577-582, 593, 612, 649-652, 673, 683, 690, 710, 845, 877, 907-916, 952, 977-978, 997-1002, 1011, 1026-1028, 1032, 1035, 1050-1051, 1132, 1184, 1226-1227, 1233, 1243, 1293-1299, 1329, 1349, 1386-1388
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         188      18  90.43%   107, 111, 135-141, 198-201, 294, 315, 439, 486, 508-511, 520
plugins/search/build_search_result.py          501      67  86.63%   246-247, 283, 286, 324, 327, 358-360, 581-592, 719, 721, 778, 785, 811, 849, 894, 910, 944-959, 1007, 1032, 1035, 1039, 1167-1168, 1177-1186, 1250, 1265, 1271, 1290-1299, 1420-1421, 1465, 1474-1476, 1533, 1581-1591
plugins/search/cop_marine.py                   263      53  79.85%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 233, 239, 243, 247, 258, 269-270, 278, 315-318, 324, 345, 349, 353, 357, 361-365, 371-374, 377-391, 408-411, 464-468, 473, 485
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 247-288
plugins/search/qssearch.py                     813      99  87.82%   415-416, 533-534, 557-558, 570-574, 631-647, 765-771, 829, 925, 932, 1003, 1024, 1027-1028, 1046, 1055-1056, 1083, 1156, 1165, 1170-1187, 1196, 1210, 1219-1222, 1262, 1345, 1368, 1441-1442, 1448, 1538, 1645-1649, 1715, 1718, 1722-1723, 1744-1747, 1759, 1781-1793, 1800, 1835-1837, 1860-1866, 1873, 1911, 1927, 1950, 1955-1956, 1971, 1977, 1987, 2071, 2075, 2086, 2110, 2123, 2131-2141, 2178-2182
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            83      17  79.52%   99-125, 169-172, 185, 227
types/__init__.py                              161      45  72.05%   67, 71, 80-84, 95-107, 135-137, 144-149, 194, 213, 216, 220, 254, 264-280, 285, 287, 309, 314, 322, 332
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             84       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              578      44  92.39%   71, 168, 201-202, 211-237, 240, 255, 335-339, 414-418, 498, 538-539, 568, 943-946, 954-955, 997, 1016-1017, 1046, 1064-1065, 1205, 1369, 1458, 1473, 1482, 1494, 1669
utils/cache.py                                  22       0  100.00%
utils/dates.py                                  57       4  92.98%   63-65, 138
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       7  81.58%   51, 53, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                         9169    1242  86.45%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
cli.py                         +3       0  +0.05%
config.py                      +1       0  +0.01%
api/product/_product.py        +5      +1  -0.27%
plugins/apis/ecmwf.py          -2       0  -0.19%
plugins/apis/usgs.py           -2      -1  +0.39%
plugins/download/aws.py       +14      +2  +0.16%
plugins/download/base.py      +13      +2  -0.09%
plugins/download/http.py       +6       0  +0.16%
TOTAL                         +38      +4  +0.01%

Results for commit: aefce77

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@eodag-bot
Copy link
Collaborator

eodag-bot commented Oct 21, 2025 •
edited
Loading

badge

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         246      10  95.93%   103-114, 628
config.py                                      454      32  92.95%   81-83, 92, 100, 104-106, 177, 188, 205-206, 735-737, 861-864, 908-909, 918-919, 953, 1033, 1054, 1062, 1092-1097, 1099
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    760      36  95.26%   342, 609, 640, 682-685, 723, 767, 801, 846-851, 875, 961, 1193, 1330, 1373, 1386, 1453, 1466-1467, 1543-1548, 1560-1563, 1894, 2140, 2203-2204, 2233-2234
api/search_result.py                           180      15  91.67%   113, 122, 129, 143, 187, 204, 301, 433, 438-439, 473, 487, 510-511, 517
api/product/__init__.py                         18       2  88.89%   60, 62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        215      18  91.63%   174, 243-244, 262-263, 371, 400, 407, 515, 535, 559-562, 571-574, 621
api/product/metadata_mapping.py                775      53  93.16%   125-127, 218-223, 244, 299-300, 388, 409, 461-462, 499, 520-523, 546, 558-559, 600, 623, 653-658, 717-722, 734, 742, 974, 1134, 1143-1147, 1164-1169, 1302, 1325, 1334, 1356, 1361, 1413, 1485, 1506, 1532, 1546, 1571, 1617, 1686, 1758
api/product/drivers/__init__.py                 11       0  100.00%
api/product/drivers/base.py                     23       0  100.00%
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 22       4  81.82%   48, 55, 68-69
plugins/manager.py                             173      18  89.60%   105-110, 145-150, 154, 192, 214, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                          101      10  90.10%   178-180, 228-229, 255-257, 287-288
plugins/apis/usgs.py                           182      25  86.26%   161, 267, 301, 343-345, 350, 378-379, 384, 414-421, 432-437, 459-465
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py             124      35  71.77%   52-54, 69-70, 142-149, 177-203, 226, 258-262, 279, 303, 319-320
plugins/authentication/base.py                  22       4  81.82%   45, 58, 81, 95
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              57       3  94.74%   68, 89, 135
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-113
plugins/crunch/filter_latest_tpl_name.py        31      20  35.48%   46-54, 64-95
plugins/crunch/filter_overlap.py                63      25  60.32%   62-65, 72-75, 81, 85, 89, 100-116, 131-157
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        395      73  81.52%   269, 303, 314-320, 361-364, 391-395, 509-511, 515, 546-547, 553-557, 588, 653-661, 725-818, 829-834, 872, 898, 943-945, 997
plugins/download/base.py                       273      39  85.71%   133, 162, 229-231, 309-310, 368-369, 411, 415-426, 440, 506, 521-525, 564-565, 590-607, 609-613, 663, 684, 706, 714
plugins/download/http.py                       551      77  86.03%   238, 280-283, 345-348, 351, 358-363, 394-396, 413, 428, 486, 521, 535, 549, 557-561, 577-582, 593, 612, 649-652, 673, 683, 690, 710, 845, 877, 907-916, 952, 977-978, 997-1002, 1011, 1026-1028, 1032, 1035, 1050-1051, 1132, 1184, 1226-1227, 1233, 1243, 1293-1299, 1329, 1349, 1386-1388
plugins/search/__init__.py                      25       0  100.00%
plugins/search/base.py                         188      18  90.43%   107, 111, 135-141, 198-201, 294, 315, 439, 486, 508-511, 520
plugins/search/build_search_result.py          501      67  86.63%   246-247, 283, 286, 324, 327, 358-360, 581-592, 719, 721, 778, 785, 811, 849, 894, 910, 944-959, 1007, 1032, 1035, 1039, 1167-1168, 1177-1186, 1250, 1265, 1271, 1290-1299, 1420-1421, 1465, 1474-1476, 1533, 1581-1591
plugins/search/cop_marine.py                   263      53  79.85%   57, 65-67, 77-78, 83, 88-89, 105, 107, 110, 176-177, 233, 239, 243, 247, 258, 269-270, 278, 315-318, 324, 345, 349, 353, 357, 361-365, 371-374, 377-391, 408-411, 464-468, 473, 485
plugins/search/creodias_s3.py                   29       1  96.55%   59
plugins/search/csw.py                          112      87  22.32%   99-100, 104-105, 113-170, 176-189, 197-229, 247-288
plugins/search/qssearch.py                     813      99  87.82%   415-416, 533-534, 557-558, 570-574, 631-647, 765-771, 829, 925, 932, 1003, 1024, 1027-1028, 1046, 1055-1056, 1083, 1156, 1165, 1170-1187, 1196, 1210, 1219-1222, 1262, 1345, 1368, 1441-1442, 1448, 1538, 1645-1649, 1715, 1718, 1722-1723, 1744-1747, 1759, 1781-1793, 1800, 1835-1837, 1860-1866, 1873, 1911, 1927, 1950, 1955-1956, 1971, 1977, 1987, 2071, 2075, 2086, 2110, 2123, 2131-2141, 2178-2182
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            83      17  79.52%   99-125, 169-172, 185, 227
types/__init__.py                              161      45  72.05%   67, 71, 80-84, 95-107, 135-137, 144-149, 194, 213, 216, 220, 254, 264-280, 285, 287, 309, 314, 322, 332
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                             84       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              578      44  92.39%   71, 168, 201-202, 211-237, 240, 255, 335-339, 414-418, 498, 538-539, 568, 943-946, 954-955, 997, 1016-1017, 1046, 1064-1065, 1205, 1369, 1458, 1473, 1482, 1494, 1669
utils/cache.py                                  22       0  100.00%
utils/dates.py                                  57       4  92.98%   63-65, 138
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       7  81.58%   51, 53, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/s3.py                                    240      12  95.00%   200-203, 245, 263, 489, 537-538, 585, 660, 686
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                         9169    1244  86.43%

Diff against develop

Filename                    Stmts    Miss  Cover
------------------------  -------  ------  -------
cli.py                         +3       0  +0.05%
config.py                      +1       0  +0.01%
api/product/_product.py        +5      +1  -0.27%
plugins/apis/ecmwf.py          -2       0  -0.19%
plugins/apis/usgs.py           -2      -1  +0.39%
plugins/download/aws.py       +14      +2  +0.16%
plugins/download/base.py      +13      +2  -0.06%
plugins/download/http.py       +6       0  +0.16%
TOTAL                         +38      +4  +0.01%

Results for commit: aefce77

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

@anesson-cs anesson-cs marked this pull request as draft October 21, 2025 13:15
anesson-cs and others added 12 commits November 4, 2025 18:37
@anesson-cs
perf: parallel dowloads
75ab1cb 
during 'download_all()', download of products are done in parallel

when a product has assets, they are downloaded in parallel
@anesson-cs
refactor: remove useless rewritings of 'download_all()' methods
3fd7400
@anesson-cs
feat: make downloads parallelization work in CLI mode
028b550
@anesson-cs
fix: deal with providers with workers limit
2145b9c
@anesson-cs
fix: display each progress bar at a different position in the shell
f779518 
before, one progress bar was permanently updated and it was unreadable

a counter of executor passings allow to determine the position of the bars
@anesson-cs
test: fix some tests
cc71ca9
@anesson-cs
refactor: use an other mechanism to identify executor origin
9f8c929 
instead of having 'in_parallel' argument to pass to EOProduct download() method, set the thread name prefix of the executor in download_all() core method
@sbrunato @anesson-cs
fix: append stream to product instead of plugin
c8a1332
@anesson-cs
fix: use stream of products during download retries
76ecdb7
@anesson-cs
test: mock stream of products in some tests
173fd96
@anesson-cs
fix: allow retry with parallel downloads
f468914
@anesson-cs
docs: add documentation about parallel downloads
f20515d
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

Parallel processing for downloading and unziping

4 participants

@anesson-cs @eodag-bot @sbrunato