Extension support implementation

[javagl]

Drafts for implementing extension support.

This addresses #109. It builds on the state from #134 , because restructuring aspects of the material is necessary for versatile, material-related extension support.

---

The current state here reflects what has been described in an earlier comment, and that I'll try to summarize here.

Internals

There now is a package de.javagl.jgltf.model.extensions that contains the infrastructure for the extension handling. This should be fairly irrelevant for clients. Most of the extension handling itself should be hidden from clients, so that clients can just use and access the extensions, without having to care too much about where they are coming from. But a quick summary:

• There is an ExtensionHandlerRegistry that allows obtaining an ExtensionHandler for a given "model class" and the extension name. For example, when there is an extension for materials, then this will be ExtensionHandler h = r.get(MaterialModel.class, "EXT_example_extension");
• The ExtensionHandler offers methods for converting between the "low-level" (JSON) structure of an extension and the "high-level" (model) representation. For illustration, it may receive some "JSON"-description of a 'texture', and convert this into a TextureModel
• The ExtensionHandlerRegistry is populated with a service loader. This means that an implementation of the ExtensionHandler can just be dropped into the classpath, and will be picked up and be available internally
• The ExtensionHandler objects are fetched when loading a glTF. When the JSON-form of the glTF is translated into the 'model'-representation, it will query any extension handler that fits the current object, and convert the JSON-form of its extensions into the model-form of the extensions.

Again: Until now, nothing of that is "visible" for clients.

Implemented extensions

The current state here contains draft implementations of several extensions that I used for checking the validity of the general approach. For each extension, there is one (typed), low-level representation of the extension structures with the name jgltf-impl-v2-<extension name>, and one project that defines the "model" structures, jgltf-model-<extension name>. (The low-level ones are generated with https://github.com/javagl/JsonModelGen , but can be generated with any other means - or simply be omitted if the implementor likes to juggle with raw JSON data...).

Right now, these extensions are

• jgltf-impl-v2-khr-lights-punctual and jgltf-model-khr-lights-punctual
• jgltf-impl-v2-khr-materials-clearcoat and jgltf-model-khr-materials-clearcoat
• jgltf-impl-v2-khr-materials-variants and jgltf-model-khr-materials-variants
• jgltf-impl-v2-khr-texture-transform and jgltf-model-khr-texture-transform

(Technical aside: The jgtlf-model-* projects are defining the META-INF/services/de.javagl.jgltf.model.extensions.ExtensionHandler files that are picked up by the service loader to discover the respective ExtensionHandler implementations of these extensions)

I used these extensions because they allow some baseline checks (like KHR_lights_punctual), as well as some "non-trivial" cases. Specifically, the combination of KHR_materials_clearcoat and KHR_texture_transform. The first extension may define a new texture. And this new texture may use the second extension. Getting these interdependencies right will still require a few iterations.

Usage

For clients, the usage of extensions should be fairly simple: When they have any "model" object, then they can use the getExtensionModel class to obtain possible extensions. For example, for a material that has the KHR_materials_clearcoat extension, they can just call

MaterialsClearcoatModel clearcoat = materialModel.getExtensionModel(
  "KHR_materials_clearcoat", MaterialsClearcoatModel.class);

For the example of combining extensions, I created a material that uses KHR_materials_clearcoat and KHR_texture_transform:

  "materials": [
    {
      "pbrMetallicRoughness": {
        "baseColorTexture": {
          "index": 0,
          "extensions": {
            "KHR_texture_transform": {
              "scale": [
                123,
                234
              ]
            }
          }
        }
      },
      "extensions": {
        "KHR_materials_clearcoat": {
          "clearcoatFactor": 0,
          "clearcoatRoughnessFactor": 0,
          "clearcoatTexture": {
            "index": 0,
            "extensions": {
              "KHR_texture_transform": {
                "scale": [
                  678,
                  789
                ]
              }
            }
          }
        }
      }
    }
  ],

And this can be accessed with the following snippet:

// Obtain the PbrMetallicRoughness and its base color texture info
PbrMetallicRoughnessModel pbrMetallicRoughness =
    materialModel.getPbrMetallicRoughnessModel();
TextureInfoModel baseColorTextureInfo =
    pbrMetallicRoughness.getBaseColorTextureInfoModel();

// Fetch the KHR_texture_transform extension object from
// the base color texture info
TextureTransformModel baseColorTextureTransform =
    baseColorTextureInfo.getExtensionModel("KHR_texture_transform",
        TextureTransformModel.class);
System.out.println("baseColor texture transform scale "
    + Arrays.toString(baseColorTextureTransform.getScale()));

// Fetch the KHR_materials_clearcoat extension object 
// from the material, and its clearcoat texture info
MaterialsClearcoatModel clearcoat = materialModel.getExtensionModel(
    "KHR_materials_clearcoat", MaterialsClearcoatModel.class);
TextureInfoModel clearcoatTextureInfo =
    clearcoat.getClearcoatTextureInfoModel();

// Fetch the KHR_texture_transform extension object from
// the clearcoat texture info
TextureTransformModel clearcoatTextureTransform =
    clearcoatTextureInfo.getExtensionModel(
        "KHR_texture_transform", TextureTransformModel.class);
System.out.println("clearcoat texture transform scale "
    + Arrays.toString(clearcoatTextureTransform.getScale()));

Remaining tasks

Many.

Until now, this aims at reading extensions. Some structures for writing extensions may be straightforward, but I have not yet thought through where the caveats might be.

Beyond that, there have to be many more tests. Eventually, these may boil down to "roundtrip" tests, which consist of reading a model with a certain extension, writing it back, and then expecting that the output matches the input.

Some extensions have structures that may not "nicely" fit into the approach here. I'm looking at you, draco and meshopt. Implementing one of them, as a proof of concept, could be worthwhile. If there only was a way to decode Draco or meshopt with Java...

The extension handling may require additional, higher-level functionalities. For example, it may be necessary to offer a mechanism to access "all ModelElement objects of one ModelElement object", to properly handle dependencies and usage. Details have not been investigated yet.

Once all this is wrapped up, the main chunk of work will be to create the actual extension implementations. Many of them (like all the PBR material extensions) should be simple and follow a common pattern. I'll probably try to address the more difficult ones first, to further validate the approach.

[javagl]

The last commits revolved around some related tasks:

• Functionality for writing extensions. This is basically the counterpart of what has already been done with the ExtensionHandler. When reading the raw glTF and creating the GltfModel from this, then the ExtensionHandler takes care of translating raw (JSON) structures into the corresponding ...Model classes. Now, it offers an additional function that converts the ...Model representation into the raw JSON structure, and this function is called for each model element when writing
• Part of the required functionality for a generic extension handling is the possibility to fetch arbitrary ModelElement objects from a given ModelElement. For example, when a MaterialModel contains an extension that defines a new texture, then it must be possible to obtain the corresponding TextureModel from the MaterialModel, without knowing the extension explicitly. Therefore, the ModelElement interface now defines a getReferencedModelElements function for this. Extension implementations (which are themself a ModelElement) implement this accordingly.
• Several cleanups related to the material refactoring from Material refactoring #134 . This is now an integral part of this branch, and will be part of the next release
• More consistent handling of default values. There are now very few places where any default values are assigned explicitly. When an input value was null, the value will also be null in the model. (See the comment at Material refactoring #134 (comment) )

An example for reading and writing a model that uses the KHR_materials_variants extension is shown here:

GltfModelReader r = new GltfModelReader();
GltfModel gltfModel = r.read(Paths.get("./data/SimpleVariants-IN.gltf"));

GltfModelWriter w = new GltfModelWriter();
w.writeEmbedded(gltfModel, new File("./data/SimpleVariants-OUT.gltf"));

Wait, where is the extension handling happening?

Under the hood. It's magic. And it has to be tested more thoroughly. But I hope that it's possible to make this as easy and transparent for the client as this example suggests...

[javagl]

The last commits added further support for reading and writing extensions. There are some aspects of the implementations of the ExtensionHandler that are a bit inconvenient and error-prone. The implementor is responsible for handling nested extensions, and has to take care to properly convert between the 'model' and the 'raw/impl' representation. I'll try to (preferably) simplify that, or (at least) document that somewhere more explicitly. But some baseline tests are already working: An example that combines KHR_materials_clearcoat and KHR_texture_transform (with that transform being applied to the base color texture and the clearcoat texture) can already be roundtripped.

[javagl]

There is little progress here. And even less progress that is publicly visible. But I've still occasionally been working on extension support in various forms.

One of the last "milestones" was support for EXT_meshopt_compression. And now it is, in fact, already possible to read a glTF model that uses meshopt, and write it out in its uncompressed form. The reverse direction (applying meshop) is not yet possible. But one of reasons why I addressed this extension in particular is that I assume that the necessary infrastructure for accomplishing this will be required for other extensions (like Draco, if it is ever added) as well.

Another aspect is somewhat related to #105 : The handling of references between "model elements" has to be pretty generic in order to work with (arbitrary) extensions. For example, one can create a glTF asset with KHR_materials_clearcoat that uses a clearcoat texture. This has two implications: Removing the extension object will make that texture "unused". It should be possible to figure this out, and cleanly remove the unused texture. Conversely, removing the texture will make that extension object "invalid" (because it refers to a texture that does not exist). So it has to be removed from that extension object as well. Some of this is already possible, with the new getReferencedModelElements method being a key aspect. But further structures (for clean removal and "pruning") will be necessary, and I have not yet found a "nice" solution for that.