Compile API: support for OrtModel input and write output to stream

[adrianlizarraga]

Description

• Adds API function to compile an OrtModel instance that was created with the model editor API.
  • C and C++ support.
  • No Python bindings yet until we first add the model editor API to the Python bindings.
• Adds API function to write the compiled model to a user-provided output stream.
  • C, C++, and Python support

Examples

More examples can be found in the unit tests.

C++

Compiles an OrtModel to a stream that itself writes to a file:

static OrtStatus* ORT_API_CALL MyWrite(void* stream_state, const void* buffer, size_t buffer_num_bytes) {
  std::ofstream* outfile = reinterpret_cast<std::ofstream*>(stream_state);
  outfile->write(reinterpret_cast<const char*>(buffer), buffer_num_bytes);
  return nullptr;  // No error.
}

int main(int argc, char** argv) {
  // Not shown: build OrtModel using editor API ...
  Ort::Model model(opsets);
  model.AddGraph(graph);

  Ort::SessionOptions so;
  so.AppendExecutionProvider("QNN", QnnHTPOptionsWithoutQDQOffloading());

  const ORTCHAR_T* output_model_file = ORT_TSTR("compileapi_ortmodel_ctx.onnx");
  std::ofstream outfile(output_model_file, std::ios::binary);

  // Create model compilation options that load an OrtModel and write the compiled model bytes to MyWrite function
  Ort::ModelCompilationOptions compile_options(*ort_env, so);
  compile_options.SetInputModel(model.GetConst());
  compile_options.SetOutputModelOutStream(MyWrite, reinterpret_cast<void*>(&outfile));  // Set output stream
  compile_options.SetEpContextEmbedMode(true);

  // Compile the model.
  Ort::Status status = Ort::CompileModel(*ort_env, compile_options);
  assert(status.IsOk());
  outfile.close();

  assert(std::filesystem::exists(output_model_file));  // assert model was created.
}

Motivation and Context

Provide flexibility to users of the Compile API that, like those compiling for webnn, that want to open/write files outside of ORT.

Next PR: allow loading input model from an existing file handle.

[adrianlizarraga]

Note: fixes incomplete handling of external data that is stored in memory with the tag kTensorProtoMemoryAddressTag. I ran into this when serializing an OrtModel (created via editor api) to a model proto. There's a test in this PR that triggers this scenario.

[yuslepukhin]

This is re-worked in my OrtValue initializers PR

[skottmckay]

There's also an external PR to address. What's the preferred approach? Get something in now and replace with the re-worked OrtValue initializers PR?

#24894

[skottmckay]

Does this work with both variants of OrtModel? One completely created with the model editor API and one that augments and existing model? I think it's fine to only support the former.

[skottmckay]

do we need num_bytes_written? doesn't that complicate things vs. requiring the function implementer to handle all bytes they were given?

[adrianlizarraga]

Good call. Removed.

[skottmckay]

Is this going to be a problem for the WebNN use case?

[adrianlizarraga]

I'm not sure. As far as I'm aware, onnx (protobuf) does not support files larger than 2gb, so weights would have to be external for large models. Generating external weight files would seem to be an issue for web. hmm

[skottmckay]

There's also an external PR to address. What's the preferred approach? Get something in now and replace with the re-worked OrtValue initializers PR?

#24894

[skottmckay]

does this need to support wide chars? IIRC the paths in the C API do and using std::filesystem is a simple way to do that.