diff --git a/README.md b/README.md index a768b113a..7ee2c3c04 100644 --- a/README.md +++ b/README.md @@ -44,6 +44,7 @@ Next, see our ["Hello World" example](docs/HelloWorld.md) to generate an image & - [Configuring Infinigen](docs/ConfiguringInfinigen.md) - [Extended ground-truth](docs/GroundTruthAnnotations.md) - [Generating individual assets](docs/GeneratingIndividualAssets.md) +- [Implementing new materials & assets](docs/ImplementingAssets.md) ### Coming Soon Please see our [project roadmap](https://infinigen.org/roadmap) and follow us at [https://twitter.com/PrincetonVL](https://twitter.com/PrincetonVL) for updates. diff --git a/docs/ConfiguringInfinigen.md b/docs/ConfiguringInfinigen.md index be9ea583b..b0e89004c 100644 --- a/docs/ConfiguringInfinigen.md +++ b/docs/ConfiguringInfinigen.md @@ -58,7 +58,7 @@ Here is a breakdown of what every commandline argument does, and ideas for how y - `--num_scenes` decides how many unique scenes the program will attempt to generate before terminating. Once you have removed `--specific_seed`, you can increase this to generate many scenes in sequence or in paralell. - `--configs desert.gin simple.gin` forces the command to generate a desert scene, and to do so with relatively low mesh detail, low render resolution, low render samples, and some asset types disabled. - Do `--configs snowy_mountain.gin simple.gin` to try out a different scene type (`snowy_mountain.gin` can instead be any scene_type option from `worldgen/configs/scene_types/`) - - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/tools/pipeline.py`. + - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/tools/pipeline_configs/base.gin`. - You have the option of removing `simple.gin` and specify neither of the original configs. This turns off the many detail-reduction options included in `simple.gin`, and will create scenes closer to those in our intro video, albeit at significant compute costs. Removing `simple.gin` will likely cause crashes unless using a workstation/server with large amounts of RAM and VRAM. You can find more details on optimizing scene content for performance [here](#config-overrides-for-mesh-detail-and-performance). - `--pipeline_configs local_16GB.gin monocular.gin blender_gt.gin` - `local_16GB.gin` specifies to run only a single scene at a time, and to run each task as a local python process. See [here](#configuring-available-computing-resources) for more options @@ -173,7 +173,7 @@ Most videos in the "Introducing Infinigen" launch video were made using commands ```` python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ - --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain + --cleanup big_files --warmup_sec 60000 --config high_quality_terrain ```` #### Creating large-scale stereo datasets @@ -219,7 +219,7 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen ``` python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ - --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \ + --cleanup big_files --warmup_sec 30000 --config high_quality_terrain \ --overrides camera.camera_pose_proposal.altitude=["uniform", 20, 30] ``` @@ -229,7 +229,7 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen ``` python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ - --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \ + --cleanup big_files --warmup_sec 30000 --config high_quality_terrain \ --pipeline_overrides iterate_scene_tasks.frame_range=[1,25] ``` diff --git a/docs/HelloWorld.md b/docs/HelloWorld.md index 1c01b45e5..2d4caaba4 100644 --- a/docs/HelloWorld.md +++ b/docs/HelloWorld.md @@ -40,7 +40,7 @@ Output logs should indicate what the code is working on. Use `--debug` for even We provide `tools/manage_datagen_jobs.py`, a utility which runs similar steps automatically. ``` -python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 +python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 \ --configs desert.gin simple.gin --pipeline_configs local_16GB.gin monocular.gin blender_gt.gin --pipeline_overrides LocalScheduleHandler.use_gpu=False ``` diff --git a/docs/ImplementingAssets.md b/docs/ImplementingAssets.md new file mode 100644 index 000000000..9a2cac045 --- /dev/null +++ b/docs/ImplementingAssets.md @@ -0,0 +1,188 @@ +# Implementing new materials & assets + +This guide will get you started on making your own procedural assets. It assumes you have already completed our [Installation Instructions](Installation.md). + +The workflow described in this guide requires some knowledge of python *and* the Blender UI. +- If you are familiar with Blender but not Python: + - You may have success trying out this guide anyway. You can work on your asset entirely in geometry/shaders nodes, and click one button to save them as a python file which can be run without edits by you to reproduce your asset and nodegroups. + - If you would rather not interact with code at all, please follow us for updates on an artist-friendly, zero-programming-required asset contribution system. +- If you are only familiar with Python but not Blender: + - You may have success trying out this guide anyway. If you can navigate the Blender UI (either by trial and error, or via the many great [online resources](https://www.youtube.com/watch?v=nIoXOplUvAw&t=34s)), this tutorial will help you use Blender as a powerful visual debugging tool to repeatedly test code you write in your usual IDE. + - Ultimately you can work on Infinigen by only manipulating code/text files. The best approach to get started on this at present is to read the many existing python files in the repo. We will work on better documentation and APIs for python developers. + +This guide does not cover how to add new elements to the terrain marching cubes mesh. This guide also does not cover adding different lighting, although you can use a similar nodegraph workflow as explained below to customize the existing lighting assets in the repo, such as the [sky light](../worldgen/lighting/lighting.py) or [caustics lamps](../worldgen/assets/caustics_lamp.py). + +## Setting up the Blender UI for interactive development + +Unless you intend to work solely on python/other code (and dont intend to interact much with Blender APIs) it will help you to have easy access to Infinigen via the Blender UI. + +To open the Blender UI, run the following in a terminal: +``` +cd infinigen/worldgen +$BLENDER dev_scene.blend +``` +:warning: You must use $BLENDER, which refers to the blender installation located in infinigen/blender, as it has additional dependencies installed. Using another blender installation is fine to open or inspect files, but it will not be able to run any of Blender's code or tools. + +We recommend you set your Blender UI up so you can see a Text Editor, Python Console, 3D Viewport, Geometry Nodes and a Shader Nodes window. The easiest way to do this is to complete the following steps (UI locations also marked in red in the screenshot below): +1. Click the "Geometry Nodes" tab +1. Use the dropdown in the topleft of the spreadsheet window (marked in #1 red) to change it to a Text Editor. +1. Drag up from the bottom right of the window (marked in #2 red) to split it in half vertically. Convert this new window to a Python console using the dropdown in it's top-left corner, similarly to step 2. +1. Click the New button (marked #4 in red) to add a geometry node group to the cube +1. Drag left from the bottom right corner of the geometry nodes window (similarly as in step 3) to split the window in half, and use the dropdown in the topleft of the new window to convert it into a Shader Editor. + +![Arranging Blender UI Panels](images/implementing_assets/setting_up_blender_ui_1.png) + +Once these steps are complete, you should see something similar to the following: + +![Completed result after arranging](images/implementing_assets/setting_up_blender_ui_2.png) + +You do not have to use this UI configuration all the time, but the following steps assume you have these three windows (Text Editor, 3D Viewport, Geometry Nodes) visible to you. + +## Importing Infinigen's dependencies into the Blender UI + +Finally, to import Infinigen into your Blender UI, click the 'Open' button on your `Text Editor` panel, then navigate to and open `worldgen/tools/blendscript_import_infinigen.py`. Click the play button to execute the script. + +You will need to re-run this script every time you restart Blender. + +## Generating assets/materials via Blender Python Commandline + +Now that you have imported Infinigen into Blender, you can easily access all it's assets and materials via the commandline. + +To start, we reccomend using Infinigen's sky lighting while you make your asset, so you can get a better sense of what the asset will look like in full scenes. To sample a random sky lighting, run the following two steps in your Blender console: +``` +from lighting import lighting +lighting.add_lighting() +``` + +You can use this mechanism to access any asset or python file under the `worldgen/` folder. For example run `from surfaces.scatters import grass` then `grass.apply(bpy.context.active_object)` in the Python Console window to apply our grassland scatter generator directly onto whichever object is selected & highlighted orange in your UI. The first statement imports the python script shown in `worldgen/surfaces/scatters/grass.py`, and you can use a similar statement to test out any python file under the worldgen/ folder, by replacing `surfaces.scatters` and `grass` with the relevant subfolder names and python filename. + +![White cube with procedural grass covering it](images/implementing_assets/setting_up_blender_ui_grassdemo.png) + +The Geometry Node and Shader Node windows in this screenshot show nodegraphs generated automatically by Infinigen. By default, automatically generated nodegraphs will not be neatly organized. If you want to manually inspect them in the UI, we recommend installing the Node Arrange addon (via Edit>Preferences>Addons then Search). Once installed, use it by selecting any node, and clicking the `Arrange` button shown in the right sidebar, to achieve nicely arranged nodegraphs as shown in the screenshot. + +:warning: If you edit your python code after importing it into Blender, your changes will not be used unless you manually force blender to reload the python module. Importing it a second time is not sufficient, you must either restart blender or use `importlib.reload`. + +## Implementing a new material or surface scatter + +To add a material to Infinigen, we must create a python file similar to those in `worldgen/surfaces/templates`. You are free to write such a file by hand in python using our `NodeWrangler` utility, but we recommend instead implementing your material in Blender then using our Node Transpiler to convert it to python code. + +To start, use the Blender UI to implement a material of your choice. Below, we show a simple snowy material comprised of a geometry nodegroup and a shader nodegroup applied to a highly subdivided cube. Please see the many excellent blender resources acknowledged in our README for help learning to use Blender's geometry and shader nodes. + +![Example procedural snow material](images/implementing_assets/snow_demo.png) + +### Using the Node Transpiler + +Click the folder icon in the Text Editor window, and navigate to and open `nodes/transpiler/transpiler_dev_script.py`. You should see a new script appear. Now, make sure the target object containing your material or other asset nodegraphs is selected, then click the play button in the Text Editor window to run the transpiler. It should complete in less than one second, and will result in a new script being added to the Text Editor script-selection dropdown named `generated_surface_script.py`. Here is the resultant script for the snow example material: + +``` +import bpy +import bpy +import mathutils +from numpy.random import uniform, normal, randint +from nodes.node_wrangler import Nodes, NodeWrangler +from nodes import node_utils +from nodes.color import color_category +from surfaces import surface + +def shader_material(nw: NodeWrangler): + # Code generated using version 2.6.4 of the node_transpiler + + principled_bsdf = nw.new_node(Nodes.PrincipledBSDF, + input_kwargs={'Base Color': (0.5004, 0.5149, 0.6913, 1.0000), 'Subsurface Radius': (0.0500, 0.1000, 0.1000), 'Roughness': 0.1182}) + + material_output = nw.new_node(Nodes.MaterialOutput, input_kwargs={'Surface': principled_bsdf}, attrs={'is_active_output': True}) + +def geometry_nodes(nw: NodeWrangler): + # Code generated using version 2.6.4 of the node_transpiler + + group_input = nw.new_node(Nodes.GroupInput, expose_input=[('NodeSocketGeometry', 'Geometry', None)]) + + normal = nw.new_node(Nodes.InputNormal) + + noise_texture = nw.new_node(Nodes.NoiseTexture, input_kwargs={'Scale': 2.6600, 'Detail': 0.8000, 'Roughness': 1.0000}) + + multiply = nw.new_node(Nodes.Math, input_kwargs={0: noise_texture.outputs["Fac"], 1: 0.1300}, attrs={'operation': 'MULTIPLY'}) + + noise_texture_1 = nw.new_node(Nodes.NoiseTexture, input_kwargs={'Scale': 100.0000, 'Detail': 15.0000, 'Roughness': 1.0000}) + + multiply_1 = nw.new_node(Nodes.Math, input_kwargs={0: noise_texture_1.outputs["Fac"], 1: 0.0150}, attrs={'operation': 'MULTIPLY'}) + + add = nw.new_node(Nodes.Math, input_kwargs={0: multiply, 1: multiply_1}) + + scale = nw.new_node(Nodes.VectorMath, input_kwargs={0: normal, 'Scale': add}, attrs={'operation': 'SCALE'}) + + set_position = nw.new_node(Nodes.SetPosition, + input_kwargs={'Geometry': group_input.outputs["Geometry"], 'Offset': scale.outputs["Vector"]}) + + group_output = nw.new_node(Nodes.GroupOutput, input_kwargs={'Geometry': set_position}, attrs={'is_active_output': True}) + +def apply(obj, selection=None, **kwargs): + surface.add_geomod(obj, geometry_nodes, selection=selection, attributes=[]) + surface.add_material(obj, shader_material, selection=selection) +apply(bpy.context.active_object) +``` + +The last line of this script calls `apply` on the currently selected object in the Blender UI. This means that you can test your script by creating and selecting a new object (ideally with a high resolution Subdivision Surface modifier) as shown: + +![Example setup before transpiling](images/implementing_assets/transpiler_demo.png) + +You can then click play on the `generated_surface_script` to run it, and it should reconstruct similar nodegraphs on this new object. To include your new material in the infinigen repository, edit the `transpiler_dev_script` to say `mode=write_file`, then run it again on your original nodegraph to dump a file into the `worldgen/` folder which you can then move to `surfaces/templates/mymaterial.py`. You can now import and test your material script via the commandline [as described earlier](#generating_assets_materials_via_blender_python_commandline) + +## Implementing a new 3D asset + +All asset generators in Infinigen are defined by python files in `worldgen/assets`, usually following this template: + +``` +import bpy +import numpy as np + +from placement.factory import AssetFactory +from util.math import FixedSeed + +class MyAssetFactory(AssetFactory): + + def __init__(self, factory_seed): + super().__init__(factory_seed) + with FixedSeed(factory_seed): + self.my_randomizable_parameter = np.random.uniform(0, 100) + + def create_asset(self, **kwargs) -> bpy.types.Object: + return None # define and return a blender object using the `bpy` api +``` + +You can implement the `create_asset` function however you wish so long as it produces a Blender Object as a result. Many existing assets use various different strategies, which you can use as examples: +- `assets/flower.py` uses mostly auto-generated code from transpiling a hand-designed geometry node-graph. +- `assets/grassland/grass_tuft.py` uses pure NumPy code to create and define a mesh. +- `assets/trees/generate.py` combines transpiled materials & leaves with a python-only space colonization algorithm. + +The simplest implementation for a new asset is to create a geometry nodes equivelant, transpile it similarly to as shown above, copy the code into the same file as the template shown above, and implement the `create_asset` function as shown: + +``` +from util import blender as butil + +... + +def apply(obj): + # code from the transpiler + +... +class MyAssetFactory(AssetFactory): + ... + def create_asset(self, **kwargs): + obj = butil.spawn_vert() # dummy empty object to apply on + apply(obj) + return obj +``` + +If you place the above text in a file located at `worldgen/assets/myasset.py`, you can add the following script to your Blender TextEditor and click play to repeatedly reload and test your asset generator as you continue to refine it. + +``` +import bpy +import importlib + +from assets import myasset +importlib.reload(myasset) + +seed = 0 +obj = myasset.MyAssetFactory(seed).spawn_asset() +``` \ No newline at end of file diff --git a/docs/images/implementing_assets/setting_up_blender_ui_1.png b/docs/images/implementing_assets/setting_up_blender_ui_1.png new file mode 100644 index 000000000..2ff299f3c Binary files /dev/null and b/docs/images/implementing_assets/setting_up_blender_ui_1.png differ diff --git a/docs/images/implementing_assets/setting_up_blender_ui_2.png b/docs/images/implementing_assets/setting_up_blender_ui_2.png new file mode 100644 index 000000000..da3ad4a98 Binary files /dev/null and b/docs/images/implementing_assets/setting_up_blender_ui_2.png differ diff --git a/docs/images/implementing_assets/setting_up_blender_ui_grassdemo.png b/docs/images/implementing_assets/setting_up_blender_ui_grassdemo.png new file mode 100644 index 000000000..af206c99c Binary files /dev/null and b/docs/images/implementing_assets/setting_up_blender_ui_grassdemo.png differ diff --git a/docs/images/implementing_assets/snow_demo.png b/docs/images/implementing_assets/snow_demo.png new file mode 100644 index 000000000..20b0929b4 Binary files /dev/null and b/docs/images/implementing_assets/snow_demo.png differ diff --git a/docs/images/implementing_assets/transpiler_demo.png b/docs/images/implementing_assets/transpiler_demo.png new file mode 100644 index 000000000..5fceb3927 Binary files /dev/null and b/docs/images/implementing_assets/transpiler_demo.png differ diff --git a/worldgen/core.py b/worldgen/core.py index 29a5d9c47..a86da56ec 100644 --- a/worldgen/core.py +++ b/worldgen/core.py @@ -58,7 +58,6 @@ from surfaces.scatters import ground_mushroom, slime_mold, moss, ivy, lichen, snow_layer from surfaces.scatters.utils.selection import scatter_lower, scatter_upward - from placement.factory import make_asset_collection from util import blender as butil from util import exporting @@ -87,7 +86,7 @@ def sanitize_gin_override(overrides: list): @gin.configurable def populate_scene( - output_folder, terrain, scene_seed, **params + output_folder, scene_seed, **params ): p = RandomStageExecutor(scene_seed, output_folder, params) camera = bpy.context.scene.camera @@ -276,12 +275,10 @@ def execute_tasks( bpy.context.scene.cycles.volume_preview_step_rate = 0.1 bpy.context.scene.cycles.volume_max_steps = 32 - terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets") - if Task.Coarse in task: butil.clear_scene(targets=[bpy.data.objects]) butil.spawn_empty(f'{VERSION=}') - compose_scene_func(output_folder, terrain, scene_seed) + compose_scene_func(output_folder, scene_seed) camera = cam_util.set_active_camera(*camera_id) if focal_length is not None: @@ -290,9 +287,10 @@ def execute_tasks( group_collections() if Task.Populate in task: - populate_scene(output_folder, terrain, scene_seed) + populate_scene(output_folder, scene_seed) if Task.FineTerrain in task: + terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets") terrain.fine_terrain(output_folder) group_collections() @@ -324,6 +322,7 @@ def execute_tasks( col.hide_viewport = False if Task.Render in task or Task.GroundTruth in task or Task.MeshSave in task: + terrain = Terrain(scene_seed, surface.registry, task=task, on_the_fly_asset_folder=output_folder/"assets") terrain.load_glb(output_folder) if Task.Render in task or Task.GroundTruth in task: diff --git a/worldgen/generate.py b/worldgen/generate.py index 9f840a081..42fdf15c0 100644 --- a/worldgen/generate.py +++ b/worldgen/generate.py @@ -70,23 +70,32 @@ import core as infinigen @gin.configurable -def compose_scene(output_folder, terrain, scene_seed, **params): +def compose_scene(output_folder, scene_seed, **params): p = RandomStageExecutor(scene_seed, output_folder, params) - p.run_stage('fancy_clouds', kole_clouds.add_kole_clouds) - - season = p.run_stage('season', random_season, use_chance=False) - logging.info(f'{season=}') + def add_coarse_terrain(): + terrain = Terrain(scene_seed, surface.registry, task='coarse', on_the_fly_asset_folder=output_folder/"assets") + terrain_mesh = terrain.coarse_terrain() + density.set_tag_dict(terrain.tag_dict) + return terrain, terrain_mesh + terrain, terrain_mesh = p.run_stage('terrain', add_coarse_terrain, use_chance=False, default=(None, None)) + + if terrain_mesh is None: + terrain_mesh = butil.create_noise_plane() + density.set_tag_dict({}) - terrain_mesh = p.run_stage('terrain', terrain.coarse_terrain, use_chance=False) - density.set_tag_dict(terrain.tag_dict) terrain_bvh = mathutils.bvhtree.BVHTree.FromObject(terrain_mesh, bpy.context.evaluated_depsgraph_get()) land_domain = params.get('land_domain_tags') underwater_domain = params.get('underwater_domain_tags') nonliving_domain = params.get('nonliving_domain_tags') + p.run_stage('fancy_clouds', kole_clouds.add_kole_clouds) + + season = p.run_stage('season', random_season, use_chance=False) + logging.info(f'{season=}') + def choose_forest_params(): # params to be shared between unique and instanced trees n_tree_species = randint(1, params.get("max_tree_species", 3) + 1) @@ -160,14 +169,20 @@ def add_cactus(terrain_mesh): def camera_preprocess(): camera_rigs = cam_util.spawn_camera_rigs() - scene_bvhtrees = cam_util.camera_selection_preprocessing(terrain) - return camera_rigs, scene_bvhtrees - camera_rigs, scene_bvhtrees = p.run_stage('camera_preprocess', camera_preprocess, use_chance=False) - p.run_stage('pose_cameras', lambda: cam_util.configure_cameras( - camera_rigs, scene_bvhtrees, terrain), use_chance=False) + scene_preprocessed = cam_util.camera_selection_preprocessing(terrain, terrain_mesh) + return camera_rigs, scene_preprocessed + camera_rigs, scene_preprocessed = p.run_stage('camera_preprocess', camera_preprocess, use_chance=False) + + bbox = terrain.get_bounding_box() if terrain is not None else butil.bounds(terrain_mesh) + p.run_stage( + 'pose_cameras', + lambda: cam_util.configure_cameras(camera_rigs, bbox, scene_preprocessed), + use_chance=False + ) cam = cam_util.get_camera(0, 0) - + p.run_stage('lighting', lighting.add_lighting, cam, use_chance=False) + # determine a small area of the terrain for the creatures to run around on # must happen before camera is animated, as camera may want to follow them around terrain_center, *_ = split_inview(terrain_mesh, cam=cam, @@ -196,7 +211,7 @@ def flying_creatures(): pois += p.run_stage('flying_creatures', flying_creatures, default=[]) p.run_stage('animate_cameras', lambda: cam_util.animate_cameras( - camera_rigs, scene_bvhtrees, pois=pois), use_chance=False) + camera_rigs, scene_preprocessed, pois=pois), use_chance=False) with Timer('Compute coarse terrain frustrums'): terrain_inview, *_ = split_inview(terrain_mesh, verbose=True, outofview=False, print_areas=True, diff --git a/worldgen/nodes/node_transpiler/dev_script.py b/worldgen/nodes/node_transpiler/transpiler_dev_script.py similarity index 88% rename from worldgen/nodes/node_transpiler/dev_script.py rename to worldgen/nodes/node_transpiler/transpiler_dev_script.py index 9ae9e8a89..22e1e10ef 100644 --- a/worldgen/nodes/node_transpiler/dev_script.py +++ b/worldgen/nodes/node_transpiler/transpiler_dev_script.py @@ -33,16 +33,12 @@ from nodes.node_transpiler import transpiler from nodes import node_wrangler, node_info -mode = 'write_file' +mode = 'make_script' target = 'object' dependencies = [ - 'assets.creatures.nodegroups.attach', - 'assets.creatures.nodegroups.curve', - 'assets.creatures.nodegroups.geometry', - 'assets.creatures.nodegroups.hair', - 'assets.creatures.nodegroups.math', - 'assets.creatures.nodegroups.shader', + # if your transpile target is using nodegroups taken from some python file, + # add those filepaths here so the transpiler imports from them rather than creating a duplicate definition. ] if target == 'object': diff --git a/worldgen/placement/camera.py b/worldgen/placement/camera.py index 45b9fccee..e997cb067 100644 --- a/worldgen/placement/camera.py +++ b/worldgen/placement/camera.py @@ -10,8 +10,9 @@ from random import sample import sys import warnings -from copy import deepcopy +from copy import deepcopy, copy from functools import partial +from itertools import chain import logging from pathlib import Path @@ -196,7 +197,10 @@ def terrain_camera_query(cam, terrain_bvh, terrain_tags_queries, vertexwise_min_ if dist is None: continue dists.append(dist) - if dist < min_dist or dist < vertexwise_min_dist[index]: + if ( + dist < min_dist or + (vertexwise_min_dist is not None and dist < vertexwise_min_dist[index]) + ): dists = None # means dist < min break for q in terrain_tags_queries: @@ -207,7 +211,14 @@ def terrain_camera_query(cam, terrain_bvh, terrain_tags_queries, vertexwise_min_ return dists, terrain_tags_queries_counts, n_pix @gin.configurable -def camera_pose_proposal(terrain_bvh, terrain_bbox, altitude=2, pitch=90, roll=0, headspace_retries=30): +def camera_pose_proposal( + terrain_bvh, + terrain_bbox, + altitude=2, + pitch=90, + roll=0, + headspace_retries=10 +): loc = np.random.uniform(*terrain_bbox) @@ -227,7 +238,7 @@ def camera_pose_proposal(terrain_bvh, terrain_bbox, altitude=2, pitch=90, roll=0 break logger.debug(f'camera_pose_proposal failed {headspace_retry=} due to {headspace=} {desired_alt=} {alt=}') else: # for-else triggers if no break, IE no acceptable voffset was found - logger.warning(f'camera_pose_proposal found no acceptable zoff after {headspace_retries=}') + logger.warning(f'camera_pose_proposal found no zoff for {loc=} after {headspace_retries=}') return None loc[2] = loc[2] + zoff @@ -251,19 +262,15 @@ def keep_cam_pose_proposal( min_terrain_distance=0, terrain_coverage_range=(0.5, 1), ): - # Reject cameras inside of terrain volumes - terrain_sdf = terrain.compute_camera_space_sdf(np.array(cam.location).reshape((1, 3))) + if terrain is not None: # TODO refactor + terrain_sdf = terrain.compute_camera_space_sdf(np.array(cam.location).reshape((1, 3))) + if not cam.type == 'CAMERA': cam = cam.children[0] if not cam.type == 'CAMERA': raise ValueError(f'{cam.name=} had {cam.type=}') - - if terrain_sdf <= 0: - logger.debug(f'keep_cam_pose_proposal rejects {terrain_sdf=}') - return None - bpy.context.view_layer.update() # Reject cameras too close to any placeholder vertex @@ -283,6 +290,13 @@ def keep_cam_pose_proposal( if coverage < terrain_coverage_range[0] or coverage > terrain_coverage_range[1]: return None + if terrain is None: + return 0 + + if terrain_sdf <= 0: + logger.debug(f'keep_cam_pose_proposal rejects {terrain_sdf=}') + return None + if rparams := terrain_tags_ratio: for q in rparams: if type(q) is tuple and q[0] == "closeup": @@ -329,9 +343,13 @@ def __call__(self, camera_rig, frame_curr, retry_pct, bvh): @gin.configurable def compute_base_views( cam, n_views, - terrain, terrain_bvh, terrain_bbox, terrain_tags_answers, vertexwise_min_dist, - terrain_tags_ratio, - placeholders_kd, + terrain, + terrain_bvh, + terrain_bbox, + placeholders_kd=None, + terrain_tags_answers={}, + vertexwise_min_dist=None, + terrain_tags_ratio=None, min_candidates_ratio=20, max_tries=10000, ): @@ -366,23 +384,42 @@ def compute_base_views( if len(potential_views) >= n_min_candidates: break - assert potential_views != [], "no camera view found" + + if len(potential_views) < n_views: + raise ValueError(f'Could not find {n_views} camera views') + return sorted(potential_views, reverse=True)[:n_views] @gin.configurable def camera_selection_preprocessing( - terrain, terrain_tags_ratio={}, + terrain, + terrain_mesh, + terrain_tags_ratio={}, ): + + with Timer(f'Building placeholders KDTree'): + + placeholders = list(chain.from_iterable( + c.all_objects for c in bpy.data.collections if c.name.startswith('placeholders:') + )) + placeholders = [p for p in placeholders if p.type == 'MESH'] + logging.info(f'Building placeholder kd for {len(placeholders)} objects') + placeholders_kd = butil.joined_kd(placeholders, include_origins=True) + + if terrain is None: + bvh = BVHTree.FromObject(terrain_mesh, bpy.context.evaluated_depsgraph_get()) + return dict( + terrain=None, + terrain_bvh=bvh, + placeholders_kd=placeholders_kd + ) with Timer(f'Building terrain BVHTree'): terrain_bvh, terrain_tags_answers, vertexwise_min_dist = terrain.build_terrain_bvh_and_attrs(terrain_tags_ratio.keys()) - with Timer(f'Building placeholders KDTree'): - placeholders_kd = placement.placeholder_kd() - return dict( - terrain_bvh=terrain_bvh, terrain=terrain, + terrain_bvh=terrain_bvh, terrain_tags_answers=terrain_tags_answers, vertexwise_min_dist=vertexwise_min_dist, placeholders_kd=placeholders_kd, @@ -392,16 +429,18 @@ def camera_selection_preprocessing( @gin.configurable def configure_cameras( cam_rigs, + bounding_box, scene_preprocessed, - terrain, ): bpy.context.view_layer.update() dummy_camera = spawn_camera() - terrain_bbox = terrain.get_bounding_box() - - base_views = compute_base_views(dummy_camera, n_views=len(cam_rigs), - terrain_bbox=terrain_bbox, **scene_preprocessed) + base_views = compute_base_views( + dummy_camera, + n_views=len(cam_rigs), + terrain_bbox=bounding_box, + **scene_preprocessed + ) for view, cam_rig in zip(base_views, cam_rigs): @@ -418,33 +457,42 @@ def configure_cameras( @gin.configurable def animate_cameras( - cam_rigs, scene_preprocessed, pois=None, + cam_rigs, + scene_preprocessed, + pois=None, follow_poi_chance=0.0, strict_selection=False, ): - anim_valid_pose_func = partial( keep_cam_pose_proposal, + placeholders_kd=scene_preprocessed['placeholders_kd'], terrain_bvh=scene_preprocessed['terrain_bvh'], terrain=scene_preprocessed['terrain'], - placeholders_kd=scene_preprocessed['placeholders_kd'], - terrain_tags_answers=scene_preprocessed['terrain_tags_answers'] if strict_selection else {}, vertexwise_min_dist=scene_preprocessed['vertexwise_min_dist'], + terrain_tags_answers=scene_preprocessed['terrain_tags_answers'] if strict_selection else {}, terrain_tags_ratio=scene_preprocessed['terrain_tags_ratio'] if strict_selection else {}, ) - for cam_rig in cam_rigs: + if U() < follow_poi_chance and pois is not None and len(pois): policy = animation_policy.AnimPolicyFollowObject( - target_obj=cam_rig, pois=pois, bvh=scene_preprocessed['terrain_bvh']) + target_obj=cam_rig, + pois=pois, + bvh=scene_preprocessed['terrain_bvh'] + ) else: policy = animation_policy.AnimPolicyRandomWalkLookaround() + logger.info(f'Animating {cam_rig=} using {policy=}') - animation_policy.animate_trajectory(cam_rig, scene_preprocessed['terrain_bvh'], + animation_policy.animate_trajectory( + cam_rig, + scene_preprocessed['terrain_bvh'], policy_func=policy, - validate_pose_func=anim_valid_pose_func, verbose=True, - fatal=True) + validate_pose_func=anim_valid_pose_func, + verbose=True, + fatal=True + ) @gin.configurable def save_camera_parameters(camera_pair_id, camera_ids, output_folder, frame, use_dof=False): diff --git a/worldgen/placement/placement.py b/worldgen/placement/placement.py index d6a20dfaa..f5520c148 100644 --- a/worldgen/placement/placement.py +++ b/worldgen/placement/placement.py @@ -4,7 +4,6 @@ # Authors: Alexander Raistrick -from itertools import groupby import re import logging from collections import defaultdict @@ -222,20 +221,6 @@ def populate_all(factory_class, camera, dist_cull=200, vis_cull=0, **kwargs): return results -def placeholder_kd(include=None, exclude=None): - objs = [] - if 'placeholders' in bpy.data.collections: - for c in bpy.data.collections['placeholders'].children: - classname = c.name.split('(') - if include is not None and classname not in include: - continue - if exclude is not None and classname in exclude: - continue - for obj in c.objects: - objs += [o for o in butil.iter_object_tree(obj) if o.type == 'MESH'] - - return butil.joined_kd(objs, include_origins=True) - def make_placeholders_float(placeholder_col, terrain_bvh, water): deps = bpy.context.evaluated_depsgraph_get() diff --git a/worldgen/terrain/core.py b/worldgen/terrain/core.py index e20221d0e..396afcb44 100644 --- a/worldgen/terrain/core.py +++ b/worldgen/terrain/core.py @@ -52,13 +52,18 @@ def __init__( device="cpu", main_terrain=TerrainNames.OpaqueTerrain, under_water=False, + min_distance=1 ): self.seed = seed self.device = device self.surface_registry = surface_registry self.main_terrain = main_terrain self.under_water = under_water - if Task.Coarse not in task and Task.FineTerrain not in task: return + self.min_distance = min_distance + + if Task.Coarse not in task and Task.FineTerrain not in task: + return + with Timer('Create terrain'): if asset_folder is None: if not ASSET_ENV_VAR in os.environ: @@ -297,13 +302,12 @@ def compute_camera_space_sdf(self, XYZ): return sdf - def get_bounding_box(self, min_distance=1): - self.min_distance = min_distance + def get_bounding_box(self): min_gen, max_gen = self.bounding_box if self.under_water: - max_gen[2] = min(max_gen[2], self.water_plane - min_distance) + max_gen[2] = min(max_gen[2], self.water_plane - self.min_distance) else: - min_gen[2] = max(min_gen[2], self.water_plane + min_distance) + min_gen[2] = max(min_gen[2], self.water_plane + self.min_distance) return min_gen, max_gen @gin.configurable diff --git a/worldgen/tools/asset_grid.py b/worldgen/tools/generate_individual_assets.py similarity index 100% rename from worldgen/tools/asset_grid.py rename to worldgen/tools/generate_individual_assets.py diff --git a/worldgen/tools/results/summarize.py b/worldgen/tools/results/summarize.py index 575c9f6fb..5fa4dc225 100644 --- a/worldgen/tools/results/summarize.py +++ b/worldgen/tools/results/summarize.py @@ -47,11 +47,9 @@ def summarize_folder(base_folder): output[data_type][suffix][rig][subcam][frame_str] = str(file_path.relative_to(base_folder)) max_frame = max(max_frame, int(frame_str)) - print(output.keys()) - # Rename keys - #output["Camera Pose"] = output.pop("T") - #output["Camera Intrinsics"] = output.pop("K") + output["Camera Pose"] = output.pop("T") + output["Camera Intrinsics"] = output.pop("K") mask_tag_jsons = sorted(parse_mask_tag_jsons(base_folder)) for frame in range(1, max_frame+1): diff --git a/worldgen/util/blender.py b/worldgen/util/blender.py index d339e2246..4260b9d6a 100644 --- a/worldgen/util/blender.py +++ b/worldgen/util/blender.py @@ -753,4 +753,16 @@ def count_instance(): depsgraph = bpy.context.evaluated_depsgraph_get() return len([inst for inst in depsgraph.object_instances if inst.is_instance]) - \ No newline at end of file + +def bounds(obj): + bbox = np.array(obj.bound_box) + return bbox.min(axis=0), bbox.max(axis=0) + +def create_noise_plane(size=50, cuts=10, std=3, levels=3): + bpy.ops.mesh.primitive_grid_add(size=size, x_subdivisions=cuts, y_subdivisions=cuts) + obj = bpy.context.active_object + + for v in obj.data.vertices: + v.co[2] = v.co[2] + np.random.normal(0, std) + + return modify_mesh(obj, 'SUBSURF', levels=levels) \ No newline at end of file