MAXIMUM_MEMORY is capped at 16 GB for wasm64 builds #25862
Description
Version of emscripten/emsdk:
emcc (Emscripten gcc/clang-like replacement + linker emulating GNU ld) 4.0.20 (6913738ec5371a88c4af5a80db0ab42bad3de681)
clang version 22.0.0git (https:/github.com/llvm/llvm-project d163988dd2833f28fbca8c144265108d25ae7bd2)
Target: wasm32-unknown-emscripten
Thread model: posix
InstalledDir: /home/luna/Software/emsdk/emsdk/upstream/bin
Failing command line in full:
(Copied from CMake logs, sorry about the length)
/emsdk/upstream/bin/wasm-ld -o scran.wasm -lembind-mt-rtti CMakeFiles/scran_wasm.dir/src/NumericMatrix.cpp.o CMakeFiles/scran_wasm.dir/src/cbind.cpp.o CMakeFiles/scran_wasm.dir/src/subset.cpp.o CMakeFiles/scran_wasm.dir/src/delayed.cpp.o CMakeFiles/scran_wasm.dir/src/matrix_stats.cpp.o CMakeFiles/scran_wasm.dir/src/initialize_from_arrays.cpp.o CMakeFiles/scran_wasm.dir/src/initialize_from_rds.cpp.o CMakeFiles/scran_wasm.dir/src/initialize_from_mtx.cpp.o CMakeFiles/scran_wasm.dir/src/initialize_from_hdf5.cpp.o CMakeFiles/scran_wasm.dir/src/transpose_matrix.cpp.o CMakeFiles/scran_wasm.dir/src/rds_utils.cpp.o CMakeFiles/scran_wasm.dir/src/hdf5_utils.cpp.o CMakeFiles/scran_wasm.dir/src/write_sparse_matrix_to_hdf5.cpp.o CMakeFiles/scran_wasm.dir/src/quality_control_rna.cpp.o CMakeFiles/scran_wasm.dir/src/quality_control_adt.cpp.o CMakeFiles/scran_wasm.dir/src/quality_control_crispr.cpp.o CMakeFiles/scran_wasm.dir/src/normalize_counts.cpp.o CMakeFiles/scran_wasm.dir/src/compute_clrm1_factors.cpp.o CMakeFiles/scran_wasm.dir/src/model_gene_variances.cpp.o CMakeFiles/scran_wasm.dir/src/run_pca.cpp.o CMakeFiles/scran_wasm.dir/src/mnn_correct.cpp.o CMakeFiles/scran_wasm.dir/src/scale_by_neighbors.cpp.o CMakeFiles/scran_wasm.dir/src/NeighborIndex.cpp.o CMakeFiles/scran_wasm.dir/src/run_tsne.cpp.o CMakeFiles/scran_wasm.dir/src/run_umap.cpp.o CMakeFiles/scran_wasm.dir/src/build_snn_graph.cpp.o CMakeFiles/scran_wasm.dir/src/cluster_graph.cpp.o CMakeFiles/scran_wasm.dir/src/cluster_kmeans.cpp.o CMakeFiles/scran_wasm.dir/src/score_markers.cpp.o CMakeFiles/scran_wasm.dir/src/run_singlepp.cpp.o CMakeFiles/scran_wasm.dir/src/score_gsdecon.cpp.o CMakeFiles/scran_wasm.dir/src/hypergeometric_test.cpp.o CMakeFiles/scran_wasm.dir/src/aggregate_across_cells.cpp.o CMakeFiles/scran_wasm.dir/src/get_error_message.cpp.o /__w/scran.js/scran.js/extern/installed/lib/libigraph.a /__w/scran.js/scran.js/extern/installed/lib/libhdf5.a /__w/scran.js/scran.js/extern/installed/lib/libhdf5_cpp.a /__w/scran.js/scran.js/extern/installed/lib/libigraph.a /__w/scran.js/scran.js/extern/installed/lib/libhdf5.a /emsdk/upstream/emscripten/cache/sysroot/lib/wasm64-emscripten/libz.a -L/emsdk/upstream/emscripten/cache/sysroot/lib/wasm64-emscripten -L/emsdk/upstream/emscripten/src/lib /emsdk/upstream/emscripten/cache/sysroot/lib/wasm64-emscripten/libz.a /emsdk/upstream/emscripten/cache/sysroot/lib/wasm64-emscripten/crtbegin.o -lGL-mt-getprocaddr -lal -lhtml5 -lstubs -lnoexit -lc-mt -ldlmalloc-mt -lcompiler_rt-mt -lc++-mt-noexcept -lc++abi-mt-noexcept -lsockets-mt -mllvm -combiner-global-alias-analysis=false -mllvm -enable-emscripten-sjlj -mllvm -disable-lsr -mwasm64 /tmp/tmp3f3gbm9xlibemscripten_js_symbols.so --import-memory --shared-memory --strip-debug --export=malloc --export=free --export=_emscripten_stack_alloc --export=__getTypeName --export=_embind_initialize_bindings --export=_emscripten_thread_free_data --export=_emscripten_thread_crashed --export=__wasm_call_ctors --export=_emscripten_tls_init --export=_emscripten_thread_init --export=emscripten_stack_get_current --export=_emscripten_stack_restore --export=emscripten_stack_set_limits --export=_emscripten_thread_exit --export-if-defined=__start_em_asm --export-if-defined=__stop_em_asm --export-if-defined=__start_em_lib_deps --export-if-defined=__stop_em_lib_deps --export-if-defined=__start_em_js --export-if-defined=__stop_em_js --export-table -z stack-size=2097152 --max-memory=53687091200 --initial-memory=16777216 --no-entry --no-stack-first --table-base=1 --global-base=1024
This gives:
wasm-ld: error: maximum memory too large, cannot be greater than 17179869184
Is this restriction really necessary for wasm64? Maybe browsers have some memory limit per page, but if I'm running Node on the backend, I should be free to use as much memory as my machine has available.
Currently I've been asking for 50GB, this is a scientific application that might process very large datasets. But honestly, it would be even better to have a -sMAXIMUM_MEMORY=inf option, which would allow me to re-use the same Wasm binary on HPCs with even more RAM.
FWIW I already compile a separate Wasm binary for web applications, so I can easily set a lower MAXIMUM_MEMORY in the web configuration to respect any browser limits.