Adds minor fix to handle normalmaps and blending

[wpumacay]

Just adds a small fix to handle normalmaps (supported as mjTEXROLE_NORMAL), and blending if the geom has an alpha value smaller than 0.99 (I think that's the cutoff they use for the blendfunction in the classic mujoco visualizer).

We have some assets in both mjcf and usd format, and they have normal maps to make them a little more real. Below is how one such asset looks in isaacsim (notice the letter has some indents shown by using a normalmap):

This is how it looks in the mujoco classic visualizer (they don't support normalmaps for rendering yet, unlike on their filament-based renderer I think):

Here is how it looks with mjviser without the fixes:

There were two issues:

• Blending wasn't enabled, the trimesh-pbrmaterial was using the default OPAQUE alphaMode, which caused the transparent pixels not being discarded correctly. We just patched it by assumming that if the geom has alpha < 0.99 from geom.rgba or material.rgba, then we set BLEND as alphaMode for the trimesh pbr material.
• We also extracted the normalmap from the mjmodel, similar to how the albedo texture is extracted, and assigned it to the pbr material (flipped this time).

This results in the following:

Let me know if you might need any changes, we're making a kind of agnostic version of mjviser to handle mujoco|mjwarp, newton, isaac and maniskill for a project we're making at Ai2, so we had to fix it there first 😅 .

PD: Thanks a lot for this package! 🙏 🙏 🙏, we're using it alongside mjlab for some experiments and it was a good reference for implementing our own version that is agnostic to the simulator.

[wpumacay]

oh @kevinzakka , here is the mjcf of that block we used, for you to test it on your end

block-sample.zip

[kevinzakka]

Thanks so much @wpumacay! I have 2 questions:

1. The normal-map texid isn't guarded the way the albedo one is (if texid_albedo >= 0). When a material has an albedo texture but no normal map, mat_texid[matid, mjTEXROLE_NORMAL] is -1, and _extract_texture_image would then index the tex_* arrays with -1, which wraps around to the last texture in the model rather than returning None. Should there be guard so we don't apply a stray texture as a normal map on models that have no normal map?
2. The normal map gets an extra FLIP_TOP_BOTTOM, but _extract_texture_image already flips MuJoCo's bottom-left origin to top-left. Wouldn't the extra flip leave it mirrored relative to the albedo?

[wpumacay]

Oh, will guard it then to make sure, np. Uhmm, I had to add the flip bc the extract texture function flips the image when it has 3 channels, but seems that normal maps as stored in the mjmodel shouldn't be flipped. I had weird artifacts bc of that and tracked it to be that issue 😅

[wpumacay]

Just pushed a change to guard for non existent normalmaps. Uhmm, I'm assuming that if a normal map is defined, at least the albedo has to be defined too, although in theory it could happen that a normalmap exists and no albedo map is given, right? (like having just a plain cube without an albedo texture, but having details defined by a normalmap, not sure if mujoco allows this though). Just in case I tested it with the cards.xml model from mujoco and it looks fine (no normalmaps on it)

Also, here is what I get when I don't flip the normalmap one extra time:

Seems that this is a tradeoff from parsing the mjmodel.tex_data, just checked the code and they don't internally flip any texture when copying to this buffer (on the C side), so most likely the normalmaps are fine as they are read in tex_data, but the first np.flipud in _extract_texture_image messes up the normalmap.

Here is a comparison of the normal maps saved from image_normalmap.save("..."), just added it on my end for testing and get the texture (the left is without the extra flip for the normalmap, the right is the original normalmapsfor the mjcf model, which not shown here as a 3rd image, but it's the same as saving the pil normalmap to disk):

[kevinzakka]

Thanks @wpumacay! I confirmed that you're right by verifying pixel-for-pixel: albedo is vflip=true, normal map isn't, so the extra flip lines them up. Also pushed a commit on top for alpha-texture blending and tests. Thanks!