perf: reduce memory footprint and add backpressure limit

[alambare]

Files were fully loaded into memory.

The back-pressure limit prevents the explosion of the buffer size when waiting on long pending requests.

[github-actions]

Test Results

    4 files  ±0      4 suites  ±0   3m 54s ⏱️ +7s
  673 tests ±0    670 ✅ ±0    3 💤 ±0  0 ❌ ±0 
2 692 runs  ±0  2 590 ✅ ±0  102 💤 ±0  0 ❌ ±0 

Results for commit 76bf758. ± Comparison against base commit 1bdf8a9.

[eodag-bot]

Code Coverage (Ubuntu)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         322      65  79.81%   63-64, 88, 689-753, 855-906, 910
config.py                                      452      30  93.36%   81-83, 92, 100, 104-106, 177, 188, 205-206, 729-731, 855-858, 902-903, 912-913, 947, 1027, 1086-1091, 1093
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    788      76  90.36%   337, 604, 635, 679-682, 720, 764, 798, 843-848, 874, 960, 1288-1300, 1340, 1342, 1372, 1376-1387, 1400-1406, 1491-1494, 1531-1551, 1620-1625, 1637-1640, 1658-1670, 1965-1971, 2222, 2226-2230, 2239-2241, 2285-2286, 2315-2316
api/search_result.py                           126      11  91.27%   87, 96, 103, 117, 282, 287-288, 336, 359-360, 366
api/product/__init__.py                         18       1  94.44%   62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        213      19  91.08%   72-74, 238-239, 345, 372, 473, 491, 515-518, 527-530, 585-588
api/product/metadata_mapping.py                726      63  91.32%   123-125, 256-257, 314-326, 328, 339, 364, 416-417, 454, 475-478, 501, 512-514, 615-616, 640-641, 651-654, 669-670, 827, 878, 1031, 1040-1044, 1061-1066, 1193, 1207-1227, 1247, 1252, 1405, 1419, 1444, 1490, 1551, 1574-1575, 1595-1599, 1615, 1623
api/product/drivers/__init__.py                 20       2  90.00%   33-36
api/product/drivers/base.py                     29       1  96.55%   98
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       2  90.48%   48, 55
plugins/manager.py                             173      19  89.02%   105-110, 145-150, 154, 192, 214, 218, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   165-167, 214-215, 241-243, 296-297
plugins/apis/usgs.py                           180      26  85.56%   156, 258, 292, 327-329, 334, 360-361, 366, 396-403, 414-419, 441-447, 478
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   45, 58
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/oauth.py                 15       7  53.33%   48-50, 54-57
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              53       3  94.34%   67, 88, 134
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47       8  82.98%   52-53, 69, 78-81, 83, 90-93
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        493     126  74.44%   271, 284, 353-356, 370-374, 466-468, 472, 505-506, 512-516, 547, 594, 598, 605, 635-643, 647, 682-690, 701-703, 764-852, 863-868, 921, 936-938, 941, 951-960, 968-982, 992-1015, 1022-1035, 1073, 1099, 1144-1146, 1366
plugins/download/base.py                       260      35  86.54%   129, 158, 305-306, 364-365, 407, 411-422, 436, 504-508, 538-539, 564-572, 574-578, 628, 649, 671, 679
plugins/download/creodias_s3.py                 26      10  61.54%   58-73
plugins/download/http.py                       545      99  81.83%   236, 278-281, 342-345, 348, 355-360, 391-393, 410, 425, 479, 514, 528, 542, 550-554, 570-575, 586, 604, 640-643, 664, 674, 681, 701, 779, 797-826, 833-848, 868, 898-907, 943, 968-969, 988-993, 1002, 1017-1019, 1023, 1026, 1041-1042, 1120-1121, 1125, 1177, 1223-1224, 1230, 1240, 1268, 1303, 1323, 1360-1362
plugins/download/s3rest.py                     122      26  78.69%   123, 157, 164, 199, 226-233, 236-238, 242, 253-259, 267-268, 271-275, 290-293, 302, 323-326
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         170      14  91.76%   103, 107, 131-137, 274, 295, 434, 456-459, 468
plugins/search/build_search_result.py          479      56  88.31%   246-247, 283, 286, 311, 314, 345-347, 559, 561, 565, 568, 695, 697, 758, 765, 789, 826, 877, 927-942, 990, 1015, 1018, 1022, 1143-1144, 1153-1162, 1225, 1246, 1265-1274, 1391-1392, 1437, 1446-1448, 1508, 1554
plugins/search/cop_marine.py                   254      47  81.50%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 232, 238, 242, 246, 259, 270-271, 279, 312, 316, 337, 341, 345, 349, 353-357, 363-366, 369-383, 400, 449-453, 458, 470
plugins/search/creodias_s3.py                   25       1  96.00%   51
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          208      72  65.38%   195-198, 214, 225, 229-230, 241, 246, 251, 258, 271-274, 328-329, 333, 343-349, 354, 372-376, 392-395, 403-414, 431, 433, 440-443, 445-446, 464-468, 501, 511, 522, 535, 541-556, 561
plugins/search/qssearch.py                     724      95  86.88%   416-417, 461, 519, 533, 537-543, 558-559, 571-575, 632-648, 708-720, 765-768, 839-840, 891, 910, 917, 929, 986, 1007, 1010-1011, 1020-1021, 1030-1031, 1040-1041, 1068, 1154-1159, 1163-1172, 1206, 1229, 1302-1303, 1309, 1399, 1564, 1567, 1573-1574, 1595, 1623-1635, 1642, 1674-1676, 1686-1692, 1722, 1745, 1750-1751, 1766, 1782, 1869, 1876, 1887, 1910-1915, 1924, 1933
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            79      13  83.54%   100-126, 170, 183
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33       7  78.79%   35-37, 53-55, 59, 68
rest/config.py                                  25       0  100.00%
rest/constants.py                                6       0  100.00%
rest/core.py                                   259      60  76.83%   257, 296, 311-347, 443, 480-519, 596, 626-629, 702, 709-756
rest/errors.py                                  72       5  93.06%   117, 127, 138, 163-164
rest/server.py                                 194      24  87.63%   86, 109-111, 285-290, 318, 518-520, 537-542, 571, 573, 577-578, 582-583
rest/stac.py                                   331      71  78.55%   312, 334, 386-389, 416-443, 474-476, 499, 531-532, 614-654, 676-692, 784-788, 846-847, 857-862, 872-876, 925, 1015-1017
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180       6  96.67%   225-229, 282, 285, 353, 375
rest/types/queryables.py                        64      12  81.25%   61, 71-79, 105-110, 174
rest/types/stac_search.py                      125       7  94.40%   128, 174, 189-191, 199, 203
rest/utils/__init__.py                          94      12  87.23%   101-102, 121-123, 175, 185-199
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       3  86.96%   48, 60, 62
types/__init__.py                              161      40  75.16%   67, 80-84, 95-107, 135-137, 144-149, 252, 262-278, 283, 285, 307, 312, 320, 330
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            111       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              587      44  92.50%   71, 197-198, 207-233, 236, 251, 331-335, 410-414, 433-435, 449, 541, 556, 596-597, 626, 1001-1004, 1012-1013, 1051-1052, 1099-1100, 1240, 1493, 1508, 1517, 1529
utils/cache.py                                  22       0  100.00%
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/s3.py                                    263      11  95.82%   207-208, 249, 265, 287, 509, 558-559, 622, 697, 723
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10867    1628  85.02%

Diff against develop

Filename       Stmts    Miss  Cover
-----------  -------  ------  -------
utils/s3.py      +14      -2  +1.04%
TOTAL            +14      -2  +0.04%

Results for commit: 76bf758

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results

[eodag-bot]

Code Coverage (Windows)

Filename                                     Stmts    Miss  Cover    Missing
-----------------------------------------  -------  ------  -------  ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------
__init__.py                                      8       0  100.00%
cli.py                                         322      65  79.81%   63-64, 88, 689-753, 855-906, 910
config.py                                      452      31  93.14%   81-83, 92, 100, 104-106, 177, 188, 205-206, 729-731, 855-858, 902-903, 912-913, 947, 1027, 1056, 1086-1091, 1093
crunch.py                                        5       5  0.00%    20-24
api/__init__.py                                  0       0  100.00%
api/core.py                                    788      76  90.36%   337, 604, 635, 679-682, 720, 764, 798, 843-848, 874, 960, 1288-1300, 1340, 1342, 1372, 1376-1387, 1400-1406, 1491-1494, 1531-1551, 1620-1625, 1637-1640, 1658-1670, 1965-1971, 2222, 2226-2230, 2239-2241, 2285-2286, 2315-2316
api/search_result.py                           126      11  91.27%   87, 96, 103, 117, 282, 287-288, 336, 359-360, 366
api/product/__init__.py                         18       1  94.44%   62
api/product/_assets.py                          52       5  90.38%   97, 183, 191, 194-198
api/product/_product.py                        213      19  91.08%   72-74, 238-239, 345, 372, 473, 491, 515-518, 527-530, 585-588
api/product/metadata_mapping.py                726      63  91.32%   123-125, 256-257, 314-326, 328, 339, 364, 416-417, 454, 475-478, 501, 512-514, 615-616, 640-641, 651-654, 669-670, 827, 878, 1031, 1040-1044, 1061-1066, 1193, 1207-1227, 1247, 1252, 1405, 1419, 1444, 1490, 1551, 1574-1575, 1595-1599, 1615, 1623
api/product/drivers/__init__.py                 20       2  90.00%   33-36
api/product/drivers/base.py                     29       1  96.55%   98
api/product/drivers/generic.py                   7       0  100.00%
api/product/drivers/sentinel1.py                15       0  100.00%
api/product/drivers/sentinel2.py                15       0  100.00%
plugins/__init__.py                              0       0  100.00%
plugins/base.py                                 21       3  85.71%   48, 55, 68
plugins/manager.py                             173      19  89.02%   105-110, 145-150, 154, 192, 214, 218, 242, 281-282, 382-385, 397-398
plugins/apis/__init__.py                         0       0  100.00%
plugins/apis/base.py                             4       0  100.00%
plugins/apis/ecmwf.py                           97      10  89.69%   165-167, 214-215, 241-243, 296-297
plugins/apis/usgs.py                           180      26  85.56%   156, 258, 292, 327-329, 334, 360-361, 366, 396-403, 414-419, 441-447, 478
plugins/authentication/__init__.py               6       1  83.33%   31
plugins/authentication/aws_auth.py              20       0  100.00%
plugins/authentication/base.py                  17       2  88.24%   45, 58
plugins/authentication/generic.py               16       3  81.25%   50, 55, 65
plugins/authentication/header.py                19       0  100.00%
plugins/authentication/keycloak.py              46       7  84.78%   153-156, 177-182
plugins/authentication/oauth.py                 15       7  53.33%   48-50, 54-57
plugins/authentication/openid_connect.py       232      28  87.93%   91-92, 104-122, 169, 175-203, 211, 350-353, 379, 420
plugins/authentication/qsauth.py                34       1  97.06%   91
plugins/authentication/sas_auth.py              53       3  94.34%   67, 88, 134
plugins/authentication/token.py                128       9  92.97%   180, 217, 289-290, 340-344
plugins/authentication/token_exchange.py        36      14  61.11%   75, 93-121
plugins/crunch/__init__.py                       0       0  100.00%
plugins/crunch/base.py                          10       1  90.00%   43
plugins/crunch/filter_date.py                   59      14  76.27%   52-57, 69, 78, 87, 90, 100-102, 109-111, 118
plugins/crunch/filter_latest_intersect.py       47      33  29.79%   49-54, 67-112
plugins/crunch/filter_latest_tpl_name.py        31       1  96.77%   83
plugins/crunch/filter_overlap.py                66      18  72.73%   28-30, 66-69, 76-79, 85, 93, 104-120
plugins/crunch/filter_property.py               30       5  83.33%   55-60, 63-64
plugins/download/__init__.py                     0       0  100.00%
plugins/download/aws.py                        493     126  74.44%   271, 284, 353-356, 370-374, 466-468, 472, 505-506, 512-516, 547, 594, 598, 605, 635-643, 647, 682-690, 701-703, 764-852, 863-868, 921, 936-938, 941, 951-960, 968-982, 992-1015, 1022-1035, 1073, 1099, 1144-1146, 1366
plugins/download/base.py                       260      37  85.77%   129, 158, 225-227, 305-306, 364-365, 407, 411-422, 436, 504-508, 538-539, 564-572, 574-578, 628, 649, 671, 679
plugins/download/creodias_s3.py                 26      10  61.54%   58-73
plugins/download/http.py                       545      99  81.83%   236, 278-281, 342-345, 348, 355-360, 391-393, 410, 425, 479, 514, 528, 542, 550-554, 570-575, 586, 604, 640-643, 664, 674, 681, 701, 779, 797-826, 833-848, 868, 898-907, 943, 968-969, 988-993, 1002, 1017-1019, 1023, 1026, 1041-1042, 1120-1121, 1125, 1177, 1223-1224, 1230, 1240, 1268, 1303, 1323, 1360-1362
plugins/download/s3rest.py                     122      26  78.69%   123, 157, 164, 199, 226-233, 236-238, 242, 253-259, 267-268, 271-275, 290-293, 302, 323-326
plugins/search/__init__.py                      22       0  100.00%
plugins/search/base.py                         170      14  91.76%   103, 107, 131-137, 274, 295, 434, 456-459, 468
plugins/search/build_search_result.py          479      63  86.85%   246-247, 283, 286, 311, 314, 345-347, 557-568, 695, 697, 758, 765, 789, 826, 877, 893, 927-942, 990, 1015, 1018, 1022, 1143-1144, 1153-1162, 1225, 1240, 1246, 1265-1274, 1391-1392, 1437, 1446-1448, 1508, 1554
plugins/search/cop_marine.py                   254      47  81.50%   56, 64-66, 76-77, 82, 87-88, 104, 106, 109, 175-176, 232, 238, 242, 246, 259, 270-271, 279, 312, 316, 337, 341, 345, 349, 353-357, 363-366, 369-383, 400, 449-453, 458, 470
plugins/search/creodias_s3.py                   25       1  96.00%   51
plugins/search/csw.py                          105      81  22.86%   98-99, 103-104, 112-160, 166-179, 187-219, 237-278
plugins/search/data_request_search.py          208      72  65.38%   195-198, 214, 225, 229-230, 241, 246, 251, 258, 271-274, 328-329, 333, 343-349, 354, 372-376, 392-395, 403-414, 431, 433, 440-443, 445-446, 464-468, 501, 511, 522, 535, 541-556, 561
plugins/search/qssearch.py                     724     107  85.22%   416-417, 461, 519, 533, 537-543, 558-559, 571-575, 632-648, 708-720, 765-768, 839-840, 891, 910, 917, 929, 986, 1007, 1010-1011, 1020-1021, 1030-1031, 1040-1041, 1068, 1154-1159, 1163-1172, 1206, 1229, 1302-1303, 1309, 1399, 1496-1500, 1564, 1567, 1573-1574, 1595, 1623-1635, 1642, 1674-1676, 1686-1692, 1722, 1745, 1750-1751, 1766, 1782, 1857, 1869, 1876, 1887, 1910-1915, 1924, 1932-1942, 1960, 1976-1980
plugins/search/stac_list_assets.py              25      10  60.00%   44-51, 75-85
plugins/search/static_stac_search.py            79      13  83.54%   100-126, 170, 183
rest/__init__.py                                 4       2  50.00%   21-22
rest/cache.py                                   33      22  33.33%   35-37, 44-70
rest/config.py                                  25       1  96.00%   35
rest/constants.py                                6       0  100.00%
rest/core.py                                   259     151  41.70%   156, 158, 160, 163-164, 178-188, 197-198, 204, 207, 248-298, 311-347, 379-417, 432-448, 464-473, 480-519, 536, 578-663, 702, 709-756
rest/errors.py                                  72      51  29.17%   63, 68-111, 116-119, 126-129, 137-167, 175-180, 195-201
rest/server.py                                 194     194  0.00%    18-596
rest/stac.py                                   331      76  77.04%   246, 312, 334, 386-389, 416-443, 474-476, 499, 531-532, 614-654, 676-692, 719, 784-788, 846-847, 853, 857-862, 872-876, 925, 963, 996, 1015-1017
rest/types/__init__.py                           0       0  100.00%
rest/types/collections_search.py                13      13  0.00%    18-44
rest/types/eodag_search.py                     180      20  88.89%   225-229, 262-264, 282, 285, 291, 295, 353, 371-386
rest/types/queryables.py                        64      20  68.75%   54-55, 60-63, 68-79, 105-110, 119-120, 174
rest/types/stac_search.py                      125      11  91.20%   126-128, 174, 189-191, 199, 203, 251, 254
rest/utils/__init__.py                          94      30  68.09%   72-78, 98, 101-102, 121-123, 136, 143, 168-176, 183-204
rest/utils/cql_evaluate.py                      48       5  89.58%   69, 76, 90, 97, 105
rest/utils/rfc3339.py                           23       5  78.26%   43-44, 48, 60, 62
types/__init__.py                              161      41  74.53%   67, 80-84, 95-107, 135-137, 144-149, 214, 252, 262-278, 283, 285, 307, 312, 320, 330
types/bbox.py                                   43      19  55.81%   46-61, 72-74, 85-87, 99-101, 113-115, 123
types/download_args.py                           9       0  100.00%
types/queryables.py                            111       0  100.00%
types/search_args.py                            70      18  74.29%   60-64, 71-88, 103
utils/__init__.py                              587      44  92.50%   71, 197-198, 207-233, 236, 251, 331-335, 410-414, 433-435, 449, 541, 556, 596-597, 626, 1001-1004, 1012-1013, 1051-1052, 1099-1100, 1240, 1493, 1508, 1517, 1529
utils/cache.py                                  22       0  100.00%
utils/env.py                                     3       0  100.00%
utils/exceptions.py                             47       0  100.00%
utils/free_text_search.py                       65       2  96.92%   83, 91
utils/import_system.py                          28      19  32.14%   64-78, 89-99
utils/logging.py                                28       1  96.43%   41
utils/notebook.py                               44      23  47.73%   25-29, 36-41, 58-62, 72-78, 83-87
utils/repr.py                                   38       8  78.95%   51, 53, 57, 98, 122-129
utils/requests.py                               55      29  47.27%   51-52, 64, 85-96, 107-124, 128
utils/rest.py                                   36       1  97.22%   55
utils/s3.py                                    263      11  95.82%   207-208, 249, 265, 287, 509, 558-559, 622, 697, 723
utils/stac_reader.py                           113      44  61.06%   63-85, 95-97, 101, 138, 154-159, 206-216, 226-256
TOTAL                                        10867    2051  81.13%

Diff against develop

Filename       Stmts    Miss  Cover
-----------  -------  ------  -------
utils/s3.py      +14      -1  +0.64%
TOTAL            +14      -1  +0.04%

Results for commit: 76bf758

Minimum allowed coverage is 70%

♻️ This comment has been updated with latest results