Implementing Assets Documentation (princeton-vl/infinigen_internal/pr…

…inceton-vl#64)

* Documentation draft

* Make terrain optional, remove terrain dependence from camera logic

* Fix config typos

* Improve intro and blender UI setup

* Added transpiler tutorial

* Finish minimal implementing assets docs

* Downsize images, add new doc to README

* Fix image hyperlinks

* Add testing script addition

* Fix non-terrain version

README.md

@@ -44,6 +44,7 @@ Next, see our ["Hello World" example](docs/HelloWorld.md) to generate an image &
 - [Configuring Infinigen](docs/ConfiguringInfinigen.md)
 - [Extended ground-truth](docs/GroundTruthAnnotations.md)
 - [Generating individual assets](docs/GeneratingIndividualAssets.md)
+- [Implementing new materials & assets](docs/ImplementingAssets.md)
 
 ### Coming Soon
 Please see our [project roadmap](https://infinigen.org/roadmap) and follow us at [https://twitter.com/PrincetonVL](https://twitter.com/PrincetonVL) for updates. 

docs/ConfiguringInfinigen.md

@@ -58,7 +58,7 @@ Here is a breakdown of what every commandline argument does, and ideas for how y
    - `--num_scenes` decides how many unique scenes the program will attempt to generate before terminating. Once you have removed `--specific_seed`, you can increase this to generate many scenes in sequence or in paralell. 
    - `--configs desert.gin simple.gin` forces the command to generate a desert scene, and to do so with relatively low mesh detail, low render resolution, low render samples, and some asset types disabled.
       - Do `--configs snowy_mountain.gin simple.gin` to try out a different scene type (`snowy_mountain.gin` can instead be any scene_type option from `worldgen/configs/scene_types/`)
-      - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/tools/pipeline.py`.
+      - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/tools/pipeline_configs/base.gin`.
       - You have the option of removing `simple.gin` and specify neither of the original configs. This turns off the many detail-reduction options included in `simple.gin`, and will create scenes closer to those in our intro video, albeit at significant compute costs. Removing `simple.gin` will likely cause crashes unless using a workstation/server with large amounts of RAM and VRAM. You can find more details on optimizing scene content for performance [here](#config-overrides-for-mesh-detail-and-performance).
    - `--pipeline_configs local_16GB.gin monocular.gin blender_gt.gin`
       - `local_16GB.gin` specifies to run only a single scene at a time, and to run each task as a local python process. See [here](#configuring-available-computing-resources) for more options
@@ -173,7 +173,7 @@ Most videos in the "Introducing Infinigen" launch video were made using commands
 ````
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain
+    --cleanup big_files --warmup_sec 60000 --config high_quality_terrain
 ````
 
 #### Creating large-scale stereo datasets
@@ -219,7 +219,7 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen
 ```
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \
+    --cleanup big_files --warmup_sec 30000 --config high_quality_terrain \
     --overrides camera.camera_pose_proposal.altitude=["uniform", 20, 30]
 ```
 
@@ -229,7 +229,7 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen
 ```
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \
+    --cleanup big_files --warmup_sec 30000 --config high_quality_terrain \
     --pipeline_overrides iterate_scene_tasks.frame_range=[1,25]
 ```
 

docs/HelloWorld.md

@@ -40,7 +40,7 @@ Output logs should indicate what the code is working on. Use `--debug` for even
 We provide `tools/manage_datagen_jobs.py`, a utility which runs similar steps automatically.
 
 ```
-python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0
+python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 \
 --configs desert.gin simple.gin --pipeline_configs local_16GB.gin monocular.gin blender_gt.gin --pipeline_overrides LocalScheduleHandler.use_gpu=False
 ```
 

docs/ImplementingAssets.md

@@ -0,0 +1,188 @@
+# Implementing new materials & assets
+
+This guide will get you started on making your own procedural assets. It assumes you have already completed our [Installation Instructions](Installation.md).
+
+The workflow described in this guide requires some knowledge of python *and* the Blender UI. 
+- If you are familiar with Blender but not Python:
+   - You may have success trying out this guide anyway. You can work on your asset entirely in geometry/shaders nodes, and click one button to save them as a python file which can be run without edits by you to reproduce your asset and nodegroups.
+   - If you would rather not interact with code at all, please follow us for updates on an artist-friendly, zero-programming-required asset contribution system. 
+- If you are only familiar with Python but not Blender:
+   - You may have success trying out this guide anyway. If you can navigate the Blender UI (either by trial and error, or via the many great [online resources](https://www.youtube.com/watch?v=nIoXOplUvAw&t=34s)), this tutorial will help you use Blender as a powerful visual debugging tool to repeatedly test code you write in your usual IDE. 
+   - Ultimately you can work on Infinigen by only manipulating code/text files. The best approach to get started on this at present is to read the many existing python files in the repo. We will work on better documentation and APIs for python developers. 
+
+This guide does not cover how to add new elements to the terrain marching cubes mesh. This guide also does not cover adding different lighting, although you can use a similar nodegraph workflow as explained below to customize the existing lighting assets in the repo, such as the [sky light](../worldgen/lighting/lighting.py) or [caustics lamps](../worldgen/assets/caustics_lamp.py).
+
+## Setting up the Blender UI for interactive development
+
+Unless you intend to work solely on python/other code (and dont intend to interact much with Blender APIs) it will help you to have easy access to Infinigen via the Blender UI.
+
+To open the Blender UI, run the following in a terminal:
+```
+cd infinigen/worldgen
+$BLENDER dev_scene.blend
+```
+:warning: You must use $BLENDER, which refers to the blender installation located in infinigen/blender, as it has additional dependencies installed. Using another blender installation is fine to open or inspect files, but it will not be able to run any of Blender's code or tools.
+
+We recommend you set your Blender UI up so you can see a Text Editor, Python Console, 3D Viewport, Geometry Nodes and a Shader Nodes window. The easiest way to do this is to complete the following steps (UI locations also marked in red in the screenshot below):
+1. Click the "Geometry Nodes" tab
+1. Use the dropdown in the topleft of the spreadsheet window (marked in #1 red) to change it to a Text Editor.
+1. Drag up from the bottom right of the window (marked in #2 red) to split it in half vertically. Convert this new window to a Python console using the dropdown in it's top-left corner, similarly to step 2.
+1. Click the New button (marked #4 in red) to add a geometry node group to the cube
+1. Drag left from the bottom right corner of the geometry nodes window (similarly as in step 3) to split the window in half, and use the dropdown in the topleft of the new window to convert it into a Shader Editor.   
+
+![Arranging Blender UI Panels](images/implementing_assets/setting_up_blender_ui_1.png)
+
+Once these steps are complete, you should see something similar to the following:
+
+![Completed result after arranging](images/implementing_assets/setting_up_blender_ui_2.png)
+
+You do not have to use this UI configuration all the time, but the following steps assume you have these three windows (Text Editor, 3D Viewport, Geometry Nodes) visible to you. 
+
+## Importing Infinigen's dependencies into the Blender UI
+
+Finally, to import Infinigen into your Blender UI, click the 'Open' button on your `Text Editor` panel, then navigate to and open `worldgen/tools/blendscript_import_infinigen.py`. Click the play button to execute the script. 
+
+<b>You will need to re-run this script every time you restart Blender. </b>
+
+## Generating assets/materials via Blender Python Commandline
+
+Now that you have imported Infinigen into Blender, you can easily access all it's assets and materials via the commandline.
+
+To start, we reccomend using Infinigen's sky lighting while you make your asset, so you can get a better sense of what the asset will look like in full scenes. To sample a random sky lighting, run the following two steps in your Blender console:
+```
+from lighting import lighting
+lighting.add_lighting()
+```
+
+You can use this mechanism to access any asset or python file under the `worldgen/` folder. For example run `from surfaces.scatters import grass` then `grass.apply(bpy.context.active_object)` in the Python Console window to apply our grassland scatter generator directly onto whichever object is selected & highlighted orange in your UI. The first statement imports the python script shown in `worldgen/surfaces/scatters/grass.py`, and you can use a similar statement to test out any python file under the worldgen/ folder, by replacing `surfaces.scatters` and `grass` with the relevant subfolder names and python filename.
+
+![White cube with procedural grass covering it](images/implementing_assets/setting_up_blender_ui_grassdemo.png)
+
+The Geometry Node and Shader Node windows in this screenshot show nodegraphs generated automatically by Infinigen. By default, automatically generated nodegraphs will not be neatly organized. If you want to manually inspect them in the UI, we recommend installing the Node Arrange addon (via Edit>Preferences>Addons then Search). Once installed, use it by selecting any node, and clicking the `Arrange` button shown in the right sidebar, to achieve nicely arranged nodegraphs as shown in the screenshot.
+
+:warning: If you edit your python code after importing it into Blender, your changes will not be used unless you manually force blender to reload the python module. Importing it a second time is not sufficient, you must either restart blender or use `importlib.reload`.
+
+## Implementing a new material or surface scatter
+
+To add a material to Infinigen, we must create a python file similar to those in `worldgen/surfaces/templates`. You are free to write such a file by hand in python using our `NodeWrangler` utility, but we recommend instead implementing your material in Blender then using our Node Transpiler to convert it to python code. 
+
+To start, use the Blender UI to implement a material of your choice. Below, we show a simple snowy material comprised of a geometry nodegroup and a shader nodegroup applied to a highly subdivided cube. Please see the many excellent blender resources acknowledged in our README for help learning to use Blender's geometry and shader nodes.
+
+![Example procedural snow material](images/implementing_assets/snow_demo.png)
+
+### Using the Node Transpiler
+
+Click the folder icon in the Text Editor window, and navigate to and open `nodes/transpiler/transpiler_dev_script.py`. You should see a new script appear. Now, make sure the target object containing your material or other asset nodegraphs is selected, then click the play button in the Text Editor window to run the transpiler. It should complete in less than one second, and will result in a new script being added to the Text Editor script-selection dropdown named `generated_surface_script.py`. Here is the resultant script for the snow example material:
+
+```
+import bpy
+import bpy
+import mathutils
+from numpy.random import uniform, normal, randint
+from nodes.node_wrangler import Nodes, NodeWrangler
+from nodes import node_utils
+from nodes.color import color_category
+from surfaces import surface
+
+def shader_material(nw: NodeWrangler):
+    # Code generated using version 2.6.4 of the node_transpiler
+
+    principled_bsdf = nw.new_node(Nodes.PrincipledBSDF,
+        input_kwargs={'Base Color': (0.5004, 0.5149, 0.6913, 1.0000), 'Subsurface Radius': (0.0500, 0.1000, 0.1000), 'Roughness': 0.1182})
+    
+    material_output = nw.new_node(Nodes.MaterialOutput, input_kwargs={'Surface': principled_bsdf}, attrs={'is_active_output': True})
+
+def geometry_nodes(nw: NodeWrangler):
+    # Code generated using version 2.6.4 of the node_transpiler
+
+    group_input = nw.new_node(Nodes.GroupInput, expose_input=[('NodeSocketGeometry', 'Geometry', None)])
+    
+    normal = nw.new_node(Nodes.InputNormal)
+    
+    noise_texture = nw.new_node(Nodes.NoiseTexture, input_kwargs={'Scale': 2.6600, 'Detail': 0.8000, 'Roughness': 1.0000})
+    
+    multiply = nw.new_node(Nodes.Math, input_kwargs={0: noise_texture.outputs["Fac"], 1: 0.1300}, attrs={'operation': 'MULTIPLY'})
+    
+    noise_texture_1 = nw.new_node(Nodes.NoiseTexture, input_kwargs={'Scale': 100.0000, 'Detail': 15.0000, 'Roughness': 1.0000})
+    
+    multiply_1 = nw.new_node(Nodes.Math, input_kwargs={0: noise_texture_1.outputs["Fac"], 1: 0.0150}, attrs={'operation': 'MULTIPLY'})
+    
+    add = nw.new_node(Nodes.Math, input_kwargs={0: multiply, 1: multiply_1})
+    
+    scale = nw.new_node(Nodes.VectorMath, input_kwargs={0: normal, 'Scale': add}, attrs={'operation': 'SCALE'})
+    
+    set_position = nw.new_node(Nodes.SetPosition,
+        input_kwargs={'Geometry': group_input.outputs["Geometry"], 'Offset': scale.outputs["Vector"]})
+    
+    group_output = nw.new_node(Nodes.GroupOutput, input_kwargs={'Geometry': set_position}, attrs={'is_active_output': True})
+
+def apply(obj, selection=None, **kwargs):
+    surface.add_geomod(obj, geometry_nodes, selection=selection, attributes=[])
+    surface.add_material(obj, shader_material, selection=selection)
+apply(bpy.context.active_object)
+```
+
+The last line of this script calls `apply` on the currently selected object in the Blender UI. This means that you can test your script by creating and selecting a new object (ideally with a high resolution Subdivision Surface modifier) as shown:
+
+![Example setup before transpiling](images/implementing_assets/transpiler_demo.png)
+
+You can then click play on the `generated_surface_script` to run it, and it should reconstruct similar nodegraphs on this new object. To include your new material in the infinigen repository, edit the `transpiler_dev_script` to say `mode=write_file`, then run it again on your original nodegraph to dump a file into the `worldgen/` folder which you can then move to `surfaces/templates/mymaterial.py`. You can now import and test your material script via the commandline [as described earlier](#generating_assets_materials_via_blender_python_commandline)
+
+## Implementing a new 3D asset
+
+All asset generators in Infinigen are defined by python files in `worldgen/assets`, usually following this template:
+
+```
+import bpy
+import numpy as np
+
+from placement.factory import AssetFactory
+from util.math import FixedSeed
+
+class MyAssetFactory(AssetFactory):
+
+    def __init__(self, factory_seed):
+        super().__init__(factory_seed)
+        with FixedSeed(factory_seed):
+            self.my_randomizable_parameter = np.random.uniform(0, 100)
+
+    def create_asset(self, **kwargs) -> bpy.types.Object:
+        return None # define and return a blender object using the `bpy` api
+```
+
+You can implement the `create_asset` function however you wish so long as it produces a Blender Object as a result. Many existing assets use various different strategies, which you can use as examples:
+- `assets/flower.py` uses mostly auto-generated code from transpiling a hand-designed geometry node-graph.
+- `assets/grassland/grass_tuft.py` uses pure NumPy code to create and define a mesh.
+- `assets/trees/generate.py` combines transpiled materials & leaves with a python-only space colonization algorithm.
+
+The simplest implementation for a new asset is to create a geometry nodes equivelant, transpile it similarly to as shown above, copy the code into the same file as the template shown above, and implement the `create_asset` function as shown:
+
+```
+from util import blender as butil
+
+...
+
+def apply(obj):
+    # code from the transpiler
+
+...
+class MyAssetFactory(AssetFactory):
+    ...
+    def create_asset(self, **kwargs):
+        obj = butil.spawn_vert() # dummy empty object to apply on
+        apply(obj)
+        return obj
+```
+
+If you place the above text in a file located at `worldgen/assets/myasset.py`, you can add the following script to your Blender TextEditor and click play to repeatedly reload and test your asset generator as you continue to refine it.
+
+```
+import bpy
+import importlib
+
+from assets import myasset
+importlib.reload(myasset) 
+
+seed = 0
+obj = myasset.MyAssetFactory(seed).spawn_asset()
+```

worldgen/core.py

@@ -58,7 +58,6 @@
 from surfaces.scatters import ground_mushroom, slime_mold, moss, ivy, lichen, snow_layer
 from surfaces.scatters.utils.selection import scatter_lower, scatter_upward
 
-
 from placement.factory import make_asset_collection
 from util import blender as butil
 from util import exporting
@@ -87,7 +86,7 @@ def sanitize_gin_override(overrides: list):
 
 @gin.configurable
 def populate_scene(
-    output_folder, terrain, scene_seed, **params
+    output_folder, scene_seed, **params
 ):
     p = RandomStageExecutor(scene_seed, output_folder, params)
     camera = bpy.context.scene.camera
@@ -276,12 +275,10 @@ def execute_tasks(
     bpy.context.scene.cycles.volume_preview_step_rate = 0.1
     bpy.context.scene.cycles.volume_max_steps = 32
 
-    terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets")
-
     if Task.Coarse in task:
         butil.clear_scene(targets=[bpy.data.objects])
         butil.spawn_empty(f'{VERSION=}')
-        compose_scene_func(output_folder, terrain, scene_seed)
+        compose_scene_func(output_folder, scene_seed)
 
     camera = cam_util.set_active_camera(*camera_id)
     if focal_length is not None:
@@ -290,9 +287,10 @@ def execute_tasks(
     group_collections()
 
     if Task.Populate in task:
-        populate_scene(output_folder, terrain, scene_seed)
+        populate_scene(output_folder, scene_seed)
 
     if Task.FineTerrain in task:
+        terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets")
         terrain.fine_terrain(output_folder)
 
     group_collections()
@@ -324,6 +322,7 @@ def execute_tasks(
         col.hide_viewport = False
 
     if Task.Render in task or Task.GroundTruth in task or Task.MeshSave in task:
+        terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets")
         terrain.load_glb(output_folder)
 
     if Task.Render in task or Task.GroundTruth in task: