Does the wasmfs OPFS backend work with `-sPROXY_TO_WORKER`? #20305

jspanchu · 2023-09-21T01:17:15Z

jspanchu
Sep 21, 2023

I'm just now exploring the opfs backend and looking at wasmfs tests. My goal is to write a file in OPFS from javascript inside /share (I've this part figured out.) and read that file from wasm. I've made this minimal example and it gives an error:

Here are the source files which I used:

// main.cpp
#include <cstdlib>
#include <exception>
#include <fstream>
#include <iostream>
#include <string>

#include <emscripten.h>
#include <emscripten/bind.h>
#include <emscripten/wasmfs.h>

void ReadFile(const std::string &filename) {
  std::cout << __func__ << "(" << filename << ")" << std::endl;
  backend_t opfs = wasmfs_create_opfs_backend();
  int err = wasmfs_create_directory("/share", 0777, opfs);
  if (err) {
    std::cerr << "Failed to create /share" << std::endl;
  } else {
    std::cout << "Created "
              << "/share" << std::endl;
  }
  std::ifstream f;
  try {
    f.open(filename, std::ios::in | std::ios::binary | std::ios::ate);
  } catch (std::exception &e) {
    std::cerr << "Failed to open " << filename << " Exception: " << e.what()
              << std::endl;
  }
  std::string contents;
  if (f.is_open()) {
    auto size = f.tellg();
    std::cout << "File size " << size << " bytes" << std::endl;
    contents.resize(size);
    f.seekg(0, std::ios::beg);
    f.read(contents.data(), size);
    f.close();
  } else {
    std::cerr << "Failed to open " << filename << std::endl;
  }
}

using namespace emscripten;

EMSCRIPTEN_BINDINGS(my_module) { function("ReadFile", &ReadFile); }

<!doctype html>
<html>

<head>
  <meta charset="utf-8" />
</head>

<body>
  <script type="text/javascript">
    var Module = {
      print: (function () {
        return function (text) {
          text = Array.prototype.slice.call(arguments).join(' ');
          console.info(text);
        };
      })(),
      printErr: function (text) {
        text = Array.prototype.slice.call(arguments).join(' ');
        console.error(text);
      },
      canvas: function() {
        canvas = document.createElement('canvas');
        return canvas;
      }(),
    };
  </script>
  <script type="text/javascript" src="main.js"></script>
  <input id="input-file" type="file" required accepted=".vtp, .obj, .stl, .ply">
  <script type="module">
    const el = document.getElementById("input-file");
    console.log("Loaded InputFile wasm");
    const opfsRoot = await navigator.storage.getDirectory();
    const dh = await opfsRoot.getDirectoryHandle("share", { create: true });
    el.onchange = async (ev) => {
      const file = el.files[0];
      const inStream = file.stream();
      const offset = 0;
      const fh = await dh.getFileHandle(file.name, { create: true });
      const opfsStream = await fh.createWritable();
      await inStream.pipeTo(opfsStream);
      postCustomMessage(`/share/${file.name}`);
    }
  </script>
</body>

</html>

// post.js
Module.onCustomMessage = function (message) {
    const filename = message.data.userData;
    Module.ReadFile(filename);
}

# CMakeLists.txt
cmake_minimum_required(VERSION 3.15)
project(env)

add_executable(main main.cpp)
set(emscripten_optimization_flags -g3 -O0)
target_link_options(main PRIVATE
  "-lembind"
  "-sALLOW_MEMORY_GROWTH"
  "-sWASMFS"
  "-sPROXY_TO_WORKER"
  "--post-js=${CMAKE_CURRENT_LIST_DIR}/post.js"
  "-sEXPORTED_RUNTIME_METHODS=['FS']"
  "${emscripten_optimization_flags}")
target_compile_options(main PRIVATE ${emscripten_optimization_flags})

# Copy HTML to build directory
add_custom_command(
  TARGET main
  COMMAND
    ${CMAKE_COMMAND} -E copy_if_different
      "${CMAKE_CURRENT_SOURCE_DIR}/index.html"
      $<TARGET_FILE_DIR:main>
)

From debugger, it seems like the worker goes inside fopen before the root handle is pushed inside wasmfsOPFSDirectoryHandles.allocated in __wasmfs_opfs_init_root_directory. Am I doing something wrong here?

sbc100 · 2023-09-21T01:32:28Z

sbc100
Sep 21, 2023
Maintainer

I would guess wasmfs probably doesn't wok with PROXY_TO_WORKER. That settings is fairly obscure and has some pretty limited use cases, and we don't test all settings in that mode. Are you using it in production? Can you explain how/why PROXY_TO_WORKER is what you need?

8 replies

tlively Sep 22, 2023
Maintainer

In general WasmFS + OPFS requires either pthreads or asyncify (or JSPI, i.e. asyncify=2). I believe the comment on wasmfs_create_ofps_backend should be updated to say that when you are using asyncify and not threads, then the function is safe to call.

sbc100 Sep 22, 2023
Maintainer

Should we have a callback-based version that allows the usage without either threads or asyncify?

jspanchu Sep 22, 2023
Author

I'm a a little worried if going the asyncify route starts to limit some features like 64-bit, etc. @sbc100 I like the idea of a callback-based version.

tlively Sep 22, 2023
Maintainer

Yep, you guys nerd-sniped me into working on an async version of the constructor now.

tlively Sep 22, 2023
Maintainer

See #20326.

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Does the wasmfs OPFS backend work with `-sPROXY_TO_WORKER`? #20305

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{editor}}'s edit

{{editor}}'s edit

Uh oh!

Replies: 1 comment 8 replies

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Uh oh!

{{title}}

Uh oh!

Select a reply

Uh oh!

Does the wasmfs OPFS backend work with -sPROXY_TO_WORKER? #20305

Uh oh!

Uh oh!

jspanchu Sep 21, 2023

Replies: 1 comment · 8 replies

Uh oh!

sbc100 Sep 21, 2023 Maintainer

Uh oh!

tlively Sep 22, 2023 Maintainer

Uh oh!

sbc100 Sep 22, 2023 Maintainer

Uh oh!

jspanchu Sep 22, 2023 Author

Uh oh!

tlively Sep 22, 2023 Maintainer

Uh oh!

tlively Sep 22, 2023 Maintainer

Does the wasmfs OPFS backend work with `-sPROXY_TO_WORKER`? #20305

jspanchu
Sep 21, 2023

Replies: 1 comment 8 replies

sbc100
Sep 21, 2023
Maintainer

tlively Sep 22, 2023
Maintainer

sbc100 Sep 22, 2023
Maintainer

jspanchu Sep 22, 2023
Author

tlively Sep 22, 2023
Maintainer

tlively Sep 22, 2023
Maintainer