Initial buggy 3.5 fixes

    * pip-based install script

    * Update docs and manage_datagen_jobs to use conda python

    * Copy over 3.5 nodegroup interface fixes

    * Fix duplicate Value nodegroup input kwargs

    * Implement compatibility mapping functions to catch old code using no-longer-support blender nodes

    * Handle both commandline formats in argparse

    * Update transfer attributes to ignore hidden attrs

    * Fix all Msample default_value interfaces in mingzhe materials, fix remaining StoreNamed

    * patched butil and mesh tools

    * blender_internal_attr

.gitignore

@@ -1,5 +1,5 @@
-Python-3.10.2
-Python-3.10.2.tgz
+Python-*
+Python-*.tgz
 blender
 blender.tar.xz
 Blender.app

docs/GeneratingIndividualAssets.md

@@ -13,9 +13,9 @@ Shown are three examples of using our `generate_individual_assets.py` script to
 ```bash
 cd worldgen
 mkdir outputs
-$BLENDER --background --python tools/generate_individual_assets.py -- -f CoralFactory -n 8 --save_blend
-$BLENDER --background --python tools/generate_individual_assets.py -- -f seashells -n 1 --save_blend
-$BLENDER --background --python tools/generate_individual_assets.py -- -f chunkyrock -n 1 --save_blend
+python tools/generate_individual_assets.py -- -f CoralFactory -n 8 --save_blend
+python tools/generate_individual_assets.py -- -f seashells -n 1 --save_blend
+python tools/generate_individual_assets.py -- -f chunkyrock -n 1 --save_blend
 ```
 
 <p align="center">
@@ -26,7 +26,7 @@ $BLENDER --background --python tools/generate_individual_assets.py -- -f chunkyr
 
 Running the above commands will save images and .blend files into your `outputs` folder.
 
-Please run `$BLENDER --background --python tools/generate_individual_assets.py -- --help` for a full list of commandline arguments.
+Please run `python tools/generate_individual_assets.py -- --help` for a full list of commandline arguments.
 
 The most commonly used arguments are:
 - `-f` to specify the name(s) of assets or materials to generate. `-f NAME` can specify to generate three different types of objects:

docs/GroundTruthAnnotations.md

@@ -70,7 +70,7 @@ Continuing the [Hello-World](/README.md#generate-a-scene-step-by-step) example,
 
 4. Export the geometry from blender to disk
 ```
-$BLENDER -noaudio --background --python generate.py -- --seed 0 --task mesh_save -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/saved_mesh
+$python generate.py -- --seed 0 --task mesh_save -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/saved_mesh
 ```
 5. Generate dense annotations
 ```

docs/HelloWorld.md

@@ -21,16 +21,16 @@ cd worldgen
 mkdir outputs
 
 # Generate a scene layout
-$BLENDER -noaudio --background --python generate.py -- --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/helloworld/coarse
+python generate.py -- --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/helloworld/coarse
 
 # Populate unique assets
-$BLENDER -noaudio --background --python generate.py -- --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine
+python generate.py -- --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine
 
 # Render RGB images
-$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames
+python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames
 
 # Render again for accurate ground-truth
-$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image 
+python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image 
 ```
 
 Output logs should indicate what the code is working on. Use `--debug` for even more detail. After each command completes you can inspect it's `--output_folder` for results, including running `$BLENDER outputs/helloworld/coarse/scene.blend` or similar to view blender files. We hide many meshes by default for viewport stability; to view them, click "Render" or use the UI to unhide them.

install.sh

@@ -1,116 +1,52 @@
 #!/bin/bash
 
-if ! command -v wget &> /dev/null
-then
-    echo "wget could not be found, please 'sudo apt-get install wget' or 'brew install wget'"
-    exit
-fi
-
-OS=$(uname -s)
-ARCH=$(uname -m)
-
-if [ "${OS}" = "Linux" ]; then
-    BLENDER_WGET_LINK='https://download.blender.org/release/Blender3.3/blender-3.3.1-linux-x64.tar.xz'
-    BLENDER_WGET_FILE='blender.tar.xz'
-
-    BLENDER_UNTAR_DIR='blender-3.3.1-linux-x64'
-    BLENDER_DIR='blender'
-    BLENDER_PYTHON="${BLENDER_DIR}/3.3/python/bin/python3.10"
-    BLENDER_INCLUDE="${BLENDER_DIR}/3.3/python/include/python3.10"
-    BLENDER_PACKAGES="${BLENDER_DIR}/3.3/python/lib/python3.10/site-packages"
-    BLENDER_ADDONS="${BLENDER_DIR}/3.3/scripts/addons"
-    BLENDER_EXE="${BLENDER_DIR}/blender"
-
-    NURBS_SCRIPT="setup_linux.py"
-elif [ "${OS}" = "Darwin" ]; then
-    if [ "${ARCH}" = "arm64" ]; then
-        BLENDER_WGET_LINK='https://download.blender.org/release/Blender3.3/blender-3.3.1-macos-arm64.dmg'
-        NURBS_SCRIPT="setup_macos_as.py"
-    else
-        BLENDER_WGET_LINK='https://download.blender.org/release/Blender3.3/blender-3.3.1-macos-x64.dmg'
-        NURBS_SCRIPT="setup_macos.py"
-    fi
-    if [ "${ARCH}" = "arm64" ]; then
-        HOMEBREW_PREFIX="/opt/homebrew/"
-    else
-        HOMEBREW_PREFIX="/usr/local"
-    fi
-    BLENDER_WGET_FILE='blender.dmg'
-
-    BLENDER_VOLM='/Volumes/Blender'
-    BLENDER_DIR='./Blender.app'
-    BLENDER_PYTHON="${BLENDER_DIR}/Contents/Resources/3.3/python/bin/python3.10"
-    BLENDER_INCLUDE="${BLENDER_DIR}/Contents/Resources/3.3/python/include/python3.10"
-    BLENDER_PACKAGES="${BLENDER_DIR}/Contents/Resources/3.3/python/lib/python3.10/site-packages"
-    BLENDER_ADDONS="${BLENDER_DIR}/Contents/Resources/3.3/scripts/addons"
-    BLENDER_EXE="${BLENDER_DIR}/Contents/MacOS/Blender"
-
-    export CC="${HOMEBREW_PREFIX}/opt/llvm/bin/clang"
-    export CPATH="${HOMEBREW_PREFIX}/include:${CPATH}"
-else
-    echo "Unsupported OS"
-    exit -1
-fi
 REQUIREMENTS_PATH='./requirements.txt'
-PYTHON_WGET_LINK='https://www.python.org/ftp/python/3.10.2/Python-3.10.2.tgz'
-PYTHON_WGET_FILE='Python-3.10.2.tgz'
-PYTHON_DIR='Python-3.10.2'
+PYTHON_WGET_LINK='https://www.python.org/ftp/python/3.10.9/Python-3.10.9.tgz'
+PYTHON_WGET_FILE='Python-3.10.9.tgz'
+PYTHON_DIR='Python-3.10.9'
 
 git submodule init
 git submodule update
 
-if [ ! -d "${BLENDER_DIR}" ]; then
-    # Download Blender
-    wget -O "${BLENDER_WGET_FILE}" "${BLENDER_WGET_LINK}"
-
-    # Unzip Blender
-    if [ "${OS}" = "Darwin" ]; then
-        hdiutil attach "${BLENDER_WGET_FILE}"
-        cp -r "${BLENDER_VOLM}/Blender.app" "${BLENDER_DIR}"
-        hdiutil detach "${BLENDER_VOLM}"
-    else
-        tar -xf "${BLENDER_WGET_FILE}"
-        mv "${BLENDER_UNTAR_DIR}" "${BLENDER_DIR}"
-    fi
-
-    rm "${BLENDER_WGET_FILE}"
-fi
-
-# Install llvm for MacOS
-if [ "${OS}" = "Darwin" ]; then
-    brew install llvm open-mpi libomp glm glew
-fi
-
 # Install Conda dependencies
 pip install -r "${REQUIREMENTS_PATH}"
-pip install fake-bpy-module-latest
 
 if [ ! -d "${PYTHON_DIR}" ]; then
     # Install Python include file
     wget -O "${PYTHON_WGET_FILE}" "${PYTHON_WGET_LINK}"
     tar -xf "${PYTHON_WGET_FILE}"
     rm "${PYTHON_WGET_FILE}"
 fi
-cp -r "${PYTHON_DIR}/Include/"* "${BLENDER_INCLUDE}"
 
-# Install Blender dependencies
-"${BLENDER_PYTHON}" -m ensurepip
-CFLAGS="-I$(realpath ${BLENDER_INCLUDE}) ${CFLAGS}" "${BLENDER_PYTHON}" -m pip install -r "${REQUIREMENTS_PATH}"
+# Install llvm for MacOS
+if [ "${OS}" = "Darwin" ]; then
+    arch -arm64 brew install llvm open-mpi libomp glm glew
+fi
+
+bash worldgen/tools/install/install_bnurbs.sh
 
 # Build terrain
-rm -rf ${BLENDER_PACKAGES}/terrain-*
 rm -rf *.egg-info
 rm -rf __pycache__
 rm -rf ./worldgen/terrain/build
 
 cd ./worldgen/terrain
-if [ -f "/usr/local/cuda/bin/nvcc" ]; then
-    USE_CUDA=1 bash install_terrain.sh
+bash install_terrain.sh
+python setup.py build_ext --inplace --force
+cd -
+
+# Compile process_mesh (i.e. OpenGL-based ground truth)
+cd ./process_mesh
+/usr/bin/cmake -S . -Bbuild -DCMAKE_C_COMPILER=/usr/bin/gcc -DCMAKE_BUILD_TYPE=Release
+/usr/bin/cmake --build build --target all
+./build/process_mesh -in x -out x --height 100 --width 10
+if [ $? -eq 174 ]; then
+    echo "OpenGL/EGL ground truth is working."
 else
-    USE_CUDA=0 bash install_terrain.sh
+    echo "WARNING: OpenGL/EGL is not supported on this machine. If you are running from a cluster head-node, this is likely not an issue."
 fi
-"../../${BLENDER_PYTHON}" setup.py build_ext --inplace --force
 cd -
+<<<<<<< HEAD
 
 # Build NURBS
 cd ./worldgen/assets/creatures/geometry/cpp_utils
@@ -127,4 +63,6 @@ fi
 # Build Flip Fluids addon
 if [ "$1" = "flip_fluids" ] || [ "$2" = "flip_fluids" ]; then
     bash ./worldgen/tools/install/compile_flip_fluids.sh
-fi
+fi
+=======
+>>>>>>> 84118268d (Initial buggy 3.5 fixes)

requirements.txt

@@ -1,3 +1,4 @@
+bpy==3.6.0
 gin-config>=0.5.0
 tqdm
 opencv-python<=4.8.0.74

worldgen/assets/boulder.py

@@ -126,7 +126,7 @@ def geo_extrusion(nw: NodeWrangler, extrude_scale=1):
             tops.append(top)
 
         geometry = nw.new_node(Nodes.StoreNamedAttribute,
-                               [geometry, 'top', None, reduce(lambda *xs: nw.boolean_math('OR', *xs), tops)])
+            input_kwargs={'Geometry': geometry, 'Name': 'top', 'Value': reduce(lambda *xs: nw.boolean_math('OR', *xs), tops)})
         nw.new_node(Nodes.GroupOutput, input_kwargs={"Geometry": geometry})
 
     def create_asset(self, i, placeholder, face_size=0.01, distance=0, **params):

worldgen/assets/corals/star.py

@@ -53,8 +53,8 @@ def geo_separate_faces(nw: NodeWrangler):
         scale = nw.uniform(.9, 1.2)
         geometry = nw.new_node(Nodes.ScaleElements, [geometry, None, scale])
         geometry = nw.new_node(Nodes.StoreNamedAttribute,
-                               [geometry, 'custom_normal', nw.new_node(Nodes.InputNormal)],
-                               attrs={'data_type': 'FLOAT_VECTOR'})
+            input_kwargs={'Geometry': geometry, 'Name': 'custom_normal', 'Value': nw.new_node(Nodes.InputNormal)},
+            attrs={'data_type': 'FLOAT_VECTOR'})
         nw.new_node(Nodes.GroupOutput, input_kwargs={'Geometry': geometry})
 
     @staticmethod
@@ -65,7 +65,8 @@ def geo_flower(nw: NodeWrangler, size, resolution, anchor):
         normal = nw.new_node(Nodes.NamedAttribute, ['custom_normal'], attrs={'data_type': 'FLOAT_VECTOR'})
         geometry = nw.new_node(Nodes.SetPosition, [geometry, None, None, nw.scale(offset, normal)])
         outer = nw.boolean_math('AND', nw.compare('GREATER_THAN', t, .4), nw.compare('LESS_THAN', t, .6))
-        geometry = nw.new_node(Nodes.StoreNamedAttribute, [geometry, 'outermost', None, outer])
+        geometry = nw.new_node(Nodes.StoreNamedAttribute, 
+            input_kwargs={'Geometry': geometry, 'Name': 'outermost', 'Value': outer})
         nw.new_node(Nodes.GroupOutput, input_kwargs={'Geometry': geometry})
 
     def create_asset(self, face_size=0.01, **params):

worldgen/assets/creatures/generate.py

@@ -194,6 +194,8 @@ def rig():
 
     if adaptive_resolution and smooth_attrs:
         for attr in joined.data.attributes.keys():
+            if butil.blender_internal_attr(attr):
+                continue
             logger.debug(f'Smoothing attr {attr}')
             surface.smooth_attribute(joined, attr, iters=10)
 

worldgen/assets/creatures/hair.py

@@ -111,11 +111,15 @@ def nodegroup_decode_noise(nw: NodeWrangler):
     map_range_1 = nw.new_node(Nodes.MapRange,
         input_kwargs={'Value': noise_texture.outputs["Fac"], 3: separate_xyz.outputs["X"], 4: separate_xyz.outputs["Y"]})
 
-    transfer_attribute = nw.new_node(Nodes.TransferAttribute,
-        input_kwargs={'Source': group_input.outputs["Source"], 2: map_range_1.outputs["Result"], 'Source Position': group_input.outputs["Source Position"]})
-
+    transfer_attribute = nw.new_node(Nodes.SampleNearestSurface,
+        input_kwargs={
+            'Mesh': group_input.outputs["Source"], 
+            'Value': map_range_1.outputs["Result"], 
+            'Sample Position': group_input.outputs["Source Position"]
+        })
+
     group_output = nw.new_node(Nodes.GroupOutput,
-        input_kwargs={'Attribute': transfer_attribute.outputs[1]})
+        input_kwargs={'Attribute': (transfer_attribute, 'Value')})
 
 @node_utils.to_nodegroup('nodegroup_hair_grooming', singleton=True, type='GeometryNodeTree')
 def nodegroup_hair_grooming(nw: NodeWrangler):
@@ -186,7 +190,7 @@ def nodegroup_hair_grooming(nw: NodeWrangler):
         input_kwargs={'Value': spline_parameter.outputs["Factor"], 3: group_input.outputs["Root Radius"], 4: 0.0})
 
     set_curve_radius = nw.new_node(Nodes.SetCurveRadius,
-        input_kwargs={'Curve': snaprootstosurface, 'Radius': map_range.outputs["Result"]})
+        input_kwargs={'Curves': snaprootstosurface, 'Radius': map_range.outputs["Result"]})
 
     group_output = nw.new_node(Nodes.GroupOutput,
         input_kwargs={'Geometry': set_curve_radius})
@@ -222,10 +226,14 @@ def geo_transfer_hair_attributes(nw, obj, attrs):
         named_attr = nw.new_node(Nodes.NamedAttribute,
             attrs={'data_type': obj_attr.data_type},
             input_kwargs={'Name': attr_name})
-        transfer = nw.new_node(Nodes.TransferAttribute, 
+        transfer = nw.new_node(Nodes.SampleNearestSurface, 
             attrs={'data_type': obj_attr.data_type},
-            input_kwargs={'Source': object_info.outputs['Geometry'], "Attribute": named_attr, 'Source Position': hairposition})
-        attrs_out[attr_name] = (transfer, 'Attribute')
+            input_kwargs={
+                'Mesh': object_info.outputs['Geometry'], 
+                "Value": named_attr, 
+                'Sample Position': hairposition
+            })
+        attrs_out[attr_name] = (transfer, 'Value')
 
     nw.new_node(Nodes.GroupOutput, input_kwargs={
         'Geometry': group_input.outputs['Geometry'], **attrs_out})
@@ -329,7 +337,7 @@ def nodegroup_transfer_uvs_to_curves_vec3(nw: NodeWrangler):
         input_kwargs={
             'Geometry': group_input.outputs["Geometry"], 
             'Name': group_input.outputs['to_attr'], 
-            2: transfer_attribute.outputs["Attribute"]},
+            'Value': transfer_attribute.outputs["Attribute"]},
         attrs={'data_type': 'FLOAT_VECTOR', 'domain': 'CURVE'})
 
     group_output = nw.new_node(Nodes.GroupOutput,

worldgen/assets/creatures/nodegroups/hair.py

@@ -24,8 +24,8 @@ def nodegroup_comb_direction(nw: NodeWrangler):
 
     normal = nw.new_node(Nodes.InputNormal)
 
-    surface_normal = nw.new_node(Nodes.TransferAttribute,
-        input_kwargs={'Source': group_input.outputs["Surface"], 1: normal, 'Source Position': group_input.outputs["Root Positiion"]},
+    surface_normal = nw.new_node(Nodes.SampleNearestSurface,
+        input_kwargs={'Mesh': group_input.outputs["Surface"], 'Value': normal, 'Sample Position': group_input.outputs["Root Positiion"]},
         label='Surface Normal',
         attrs={'data_type': 'FLOAT_VECTOR'})
 
@@ -45,25 +45,28 @@ def nodegroup_comb_direction(nw: NodeWrangler):
         input_kwargs={0: subtract.outputs["Vector"]},
         attrs={'operation': 'NORMALIZE'})
 
-    skeleton_tangent = nw.new_node(Nodes.TransferAttribute,
-        input_kwargs={'Source': group_input.outputs["Surface"], 1: normalize.outputs["Vector"], 'Source Position': group_input.outputs["Root Positiion"]},
-        label='Skeleton Tangent',
+    skeleton_tangent = nw.new_node(Nodes.SampleNearestSurface,
+        input_kwargs={'Mesh': group_input.outputs["Surface"], 'Value': normalize.outputs["Vector"], 'Sample Position': group_input.outputs["Root Positiion"]},
         attrs={'data_type': 'FLOAT_VECTOR'})
 
     cross_product = nw.new_node(Nodes.VectorMath,
-        input_kwargs={0: surface_normal.outputs["Attribute"], 1: skeleton_tangent.outputs["Attribute"]},
+        input_kwargs={0: surface_normal, 1: skeleton_tangent},
         attrs={'operation': 'CROSS_PRODUCT'})
 
     cross_product_1 = nw.new_node(Nodes.VectorMath,
-        input_kwargs={0: surface_normal.outputs["Attribute"], 1: cross_product.outputs["Vector"]},
+        input_kwargs={0: surface_normal, 1: cross_product.outputs["Vector"]},
         attrs={'operation': 'CROSS_PRODUCT'})
 
     normalize_1 = nw.new_node(Nodes.VectorMath,
         input_kwargs={0: cross_product_1.outputs["Vector"]},
         attrs={'operation': 'NORMALIZE'})
 
     group_output = nw.new_node(Nodes.GroupOutput,
-        input_kwargs={'Combing Direction': normalize_1.outputs["Vector"], 'Surface Normal': surface_normal.outputs["Attribute"], 'Skeleton Tangent': skeleton_tangent.outputs["Attribute"]})
+        input_kwargs={
+            'Combing Direction': normalize_1.outputs["Vector"], 
+            'Surface Normal': surface_normal.outputs["Attribute"], 
+            'Skeleton Tangent': skeleton_tangent
+        })
 
 @node_utils.to_nodegroup('nodegroup_hair_position', singleton=True, type='GeometryNodeTree')
 def nodegroup_hair_position(nw: NodeWrangler):
@@ -82,20 +85,23 @@ def nodegroup_hair_position(nw: NodeWrangler):
         input_kwargs={0: index, 1: spline_length.outputs["Point Count"]},
         attrs={'operation': 'SNAP'})
 
-    hair_root_position = nw.new_node(Nodes.TransferAttribute,
-        input_kwargs={'Source': group_input.outputs["Curves"], 1: position, 'Index': snap},
+    hair_root_position = nw.new_node(Nodes.SampleIndex,
+        input_kwargs={'Geometry': group_input.outputs["Curves"], 'Value': position, 'Index': snap},
         label='Hair Root Position',
-        attrs={'data_type': 'FLOAT_VECTOR', 'mapping': 'INDEX'})
+        attrs={'data_type': 'FLOAT_VECTOR'})
 
     position_1 = nw.new_node(Nodes.InputPosition)
 
     relative_position = nw.new_node(Nodes.VectorMath,
-        input_kwargs={0: position_1, 1: hair_root_position.outputs["Attribute"]},
+        input_kwargs={0: position_1, 1: hair_root_position},
         label='Relative Position',
         attrs={'operation': 'SUBTRACT'})
-    
+
     group_output = nw.new_node(Nodes.GroupOutput,
-        input_kwargs={'Root Position': hair_root_position.outputs["Attribute"], 'Relative Position': relative_position.outputs["Vector"]})
+        input_kwargs={
+            'Root Position': hair_root_position, 
+            'Relative Position': relative_position.outputs["Vector"]
+        })
 
 @node_utils.to_nodegroup('nodegroup_comb_hairs', singleton=True, type='GeometryNodeTree')
 def nodegroup_comb_hairs(nw: NodeWrangler):

worldgen/assets/creatures/parts/leg.py

@@ -230,7 +230,7 @@ def nodegroup_insect_leg(nw: NodeWrangler):
         input_kwargs={'Skin Mesh': simple_tube_v2.outputs["Geometry"], 'Skeleton Curve': simple_tube_v2.outputs["Skeleton Curve"], 'Coord 0': (0.0, 0.0, 0.0), 'Coord 1': (0.01, 0.0, 0.0), 'Coord 2': (0.35, 0.0, 0.0), 'StartRad, EndRad, Fullness': combine_xyz, 'ProfileHeight, StartTilt, EndTilt': (0.73, 0.0, 0.0)})
 
     trim_curve = nw.new_node(Nodes.TrimCurve,
-        input_kwargs={'Curve': simple_tube_v2.outputs["Skeleton Curve"], 1: 0.4892, 2: 0.725})
+        input_kwargs={'Curve': simple_tube_v2.outputs["Skeleton Curve"], 'Start': 0.4892, 'End': 0.725})
 
     resample_curve = nw.new_node(Nodes.ResampleCurve,
         input_kwargs={'Curve': trim_curve, 'Count': 4})