Rendering improvements (princeton-vl/infinigen_internal/princeton-vl#39)

* Tweak render_video_final

* Remove random config choosing from core.py

* Create tools/pipeline_configs/base.gin, move scenetype distribution configs into it

* Create noshortrender config to test on IONIC

* Implement slurm niceness override, add it to render_video_final.sh

* Only include camera 0 in parse_video output

* Read slurm partition from ENVVAR by default

* Fix config postprocess

* Fix slurm envvar

* Typo fixes

* Use roundrobin by default

* Rendering tweaks

* Change trailer.gin to video.gin with 720p res

* Fix niceness

* Set exclude_nodes list via envvar, move niceness configs into slurm.gin

* Create render_video_720p.sh, start off experimental.gin but more needs adding

* Add dryrun options

* Fix --override vs --overrides

* Move legacy task.Fine

* Retool upload func

* Add slurm_1h and stereo config

* Rendering & typo fixes

* Update render script and slurm.gin mem amounts

* Fix excluded gpus

* Add queues stats to wandb, add pandas to requirements.txt

* Fix num_concurrent reset 24h later

* Dont keep working on scenes which have had a fatal crash

* Add new timeout message to error parsing

* Fix overly nested upload dirs

* Add thread limit to local jobs

docs/CHANGELOG.md

@@ -1,3 +1,3 @@
 v1.0.0 - Beta code release <br>
-v1.0.1 - BSD-3 license, expanded ground-truth docs, show line-credits, miscellaneous fixes
+v1.0.1 - BSD-3 license, expanded ground-truth docs, show line-credits, miscellaneous fixes <br>
 v1.0.2 - New documentation, plant improvements, disk and reproducibility improvements

docs/ConfiguringInfinigen.md

@@ -39,7 +39,7 @@ If you find a useful and related combination of these commandline overrides, you
 
 Our `generate.py` driver always loads [`worldgen/configs/base.gin`][../worldgen/configs/base.gin], and you can inspect / modify this file to see many common and useful gin override options.
 
-`generate.py` also expects that one file from (configs/scene_types/)[worldgen/config/scene_types] will be loaded, and if one is not specified on the commandline it will choose one randomly according to the keys and weights in `worldgen/core.py`. These scene_type configs contain gin overrides designed to encode the semantic constraints of real natural habitats (e.g. `worldgen/scene_types/desert.gin` causes sand to appear and cacti to be more likely).
+`generate.py` also expects that one file from (configs/scene_types/)[worldgen/config/scene_types] will be loaded. These scene_type configs contain gin overrides designed to encode the semantic constraints of real natural habitats (e.g. `worldgen/scene_types/desert.gin` causes sand to appear and cacti to be more likely).
 
 ### Moving beyond "Hello World"
 
@@ -58,7 +58,7 @@ Here is a breakdown of what every commandline argument does, and ideas for how y
    - `--num_scenes` decides how many unique scenes the program will attempt to generate before terminating. Once you have removed `--specific_seed`, you can increase this to generate many scenes in sequence or in paralell. 
    - `--configs desert.gin simple.gin` forces the command to generate a desert scene, and to do so with relatively low mesh detail, low render resolution, low render samples, and some asset types disabled.
       - Do `--configs snowy_mountain.gin simple.gin` to try out a different scene type (`snowy_mountain.gin` can instead be any scene_type option from `worldgen/configs/scene_types/`)
-      - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/core.py`.
+      - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/tools/pipeline.py`.
       - You have the option of removing `simple.gin` and specify neither of the original configs. This turns off the many detail-reduction options included in `simple.gin`, and will create scenes closer to those in our intro video, albeit at significant compute costs. Removing `simple.gin` will likely cause crashes unless using a workstation/server with large amounts of RAM and VRAM. You can find more details on optimizing scene content for performance [here](#config-overrides-for-mesh-detail-and-performance).
    - `--pipeline_configs local_16GB.gin monocular.gin blender_gt.gin`
       - `local_16GB.gin` specifies to run only a single scene at a time, and to run each task as a local python process. See [here](#configuring-available-computing-resources) for more options
@@ -126,7 +126,6 @@ You will also encounter configs using what we term a "registry pattern", e.g. `w
    - For example, in `base_surface_registry.gin`, `surface.registry.beach` specifies `("sand", 10)` to indicate that sand has high weight to be chosen to be assigned for the beach category. 
    - Weights are normalized by their overall sum to obtain a probability distribution. 
    - Name strings undergo lookup in the relevant source code folders, e.g. the name "sand" in a surface registry maps to `worldgen/surfaces/templates/sand.py`.
-   - The random choice among scene_type configs is itself a registry, although it is hardcoded in `core.py` currently, since the choice of what configs are loaded cannot depend on a config file. This will be improved soon.
 
 ### Config Overrides for mesh detail and performance
 
@@ -153,7 +152,7 @@ If you find yourself bottlenecked by GPU time, you should consider the following
    - Reduce `base.gin`'s `full/render_image.num_samples = 8192` or `compose_scene.generate_resolution = (1920, 1080)`. This proportionally reduces rendering FLOPS, with some diminishing returns due to BVH setup time.
    - If your GPU(s) are _underutilized_, try the reverse of these tips.
 
-Some scene type configs are also generally more expensive than others. `forest.gin` and `coral.gin` are very expensive due to dense detailed fauna, wheras `artic` and `snowy_mountain` are very cheap. Low-resource compute settings (<64GB) of RAM may only be able to handle a subset of our `worldgen/config/scene_type/` options, and you may wish to tune the ratios of scene_types by editing `worldgen/core.py`. 
+Some scene type configs are also generally more expensive than others. `forest.gin` and `coral.gin` are very expensive due to dense detailed fauna, wheras `artic` and `snowy_mountain` are very cheap. Low-resource compute settings (<64GB) of RAM may only be able to handle a subset of our `worldgen/config/scene_type/` options, and you may wish to tune the ratios of scene_types by editing `worldgen/tools/pipeline_configs/base.gin` or otherwise overriding `sample_scene_spec.config_distribution`. 
 
 ### Other `manage_datagen_jobs.py` commandline options
 
@@ -174,15 +173,15 @@ Most videos in the "Introducing Infinigen" launch video were made using commands
 ````
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain
+    --cleanup big_files --warmup_sec 60000 --config video high_quality_terrain
 ````
 
 #### Creating large-scale stereo datasets
 
 ````
 python -m tools.manage_datagen_jobs --output_folder outputs/stereo_data --num_scenes 10000 \
     --pipeline_config slurm stereo cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain
+    --cleanup big_files --warmup_sec 60000 --config high_quality_terrain
 ````
 
 #### Creating a few low-resolution images to your test changes
@@ -220,7 +219,7 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen
 ```
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 30000 --config trailer high_quality_terrain \
+    --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \
     --overrides camera.camera_pose_proposal.altitude=["uniform", 20, 30]
 ```
 
@@ -230,8 +229,8 @@ python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scen
 ```
 python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \
     --pipeline_config slurm monocular_video cuda_terrain opengl_gt \
-    --cleanup big_files --warmup_sec 30000 --config trailer high_quality_terrain \
+    --cleanup big_files --warmup_sec 30000 --config video high_quality_terrain \
     --pipeline_overrides iterate_scene_tasks.frame_range=[1,25]
 ```
 
-:bulb: This command uses `--pipeline_overrides` rather than `--overrides` since it is providing a gin override to the `manage_datagen_jobs.py` process, not some part main `generate.py` driver.
+:bulb: This command uses `--pipeline_overrides` rather than `--overrides` since it is providing a gin override to the `manage_datagen_jobs.py` process, not some part of the main `generate.py` driver.

requirements.txt

@@ -27,3 +27,5 @@ landlab==2.4.1
 scikit-learn
 psutil
 pyrender
+pytest
+pandas

worldgen/config/base.gin

@@ -130,9 +130,9 @@ flat/render_image.passes_to_save = [
 
 render_image.exposure = 1
 
-render_image.use_dof = 'IF_TARGET_SET'
+render_image.use_dof = False
 render_image.dof_aperture_fstop = 3
-render_image.motion_blur = True
+render_image.motion_blur = False
 render_image.motion_blur_shutter = 0.15
 
 compositor_postprocessing.distort = False
@@ -142,7 +142,6 @@ compose_scene.generate_resolution = (1920, 1080)
 get_sensor_coords.H = 720
 get_sensor_coords.W = 1280
 
-
 min_terrain_distance = 2
 keep_cam_pose_proposal.min_terrain_distance = %min_terrain_distance
 SphericalMesher.r_min = %min_terrain_distance

worldgen/config/experimental.gin

@@ -0,0 +1,3 @@
+# things that are not quite fully working correctly, but you can use if you please
+
+render_image.motion_blur = True # not fully supported in ground truth

worldgen/config/scene_types/snowy_mountain.gin

@@ -27,7 +27,7 @@ compose_scene.flying_creature_registry = [
 ]
 
 surfaces.templates.mountain.shader.layered_mountain = 0
-surfaces.templates.mountain.shader.snowy = 1
+surfaces.templates.mountain.shader.snowy = 0 # TODO: re-enable once terrain flickering resolved
 compose_scene.boulders_chance = 1
 
 camera.camera_pose_proposal.pitch = ("clip_gaussian", 90, 30, 90, 100)

worldgen/core.py

@@ -231,6 +231,7 @@ def execute_tasks(
     generate_resolution=(1920,1080),
     reset_assets=True,
     focal_length=None,
+    dryrun=False,
 ):
     if input_folder != output_folder:
         if reset_assets:
@@ -241,6 +242,10 @@ def execute_tasks(
         if (not os.path.islink(output_folder/"assets")) and (not (output_folder/"assets").exists()) and input_folder is not None and (input_folder/"assets").exists():
             os.symlink(input_folder/"assets", output_folder/"assets")
             # in this way, even coarse task can have input_folder to have pregenerated on-the-fly assets (e.g., in last run) to speed up developing
+
+    if dryrun:
+        return
+
     if Task.Coarse not in task:
         with Timer('Reading input blendfile'):
             bpy.ops.wm.open_mainfile(filepath=str(input_folder / 'scene.blend'))
@@ -287,9 +292,6 @@ def execute_tasks(
     if Task.Populate in task:
         populate_scene(output_folder, terrain, scene_seed)
 
-    if Task.Fine in task:
-        raise RuntimeError(f'{task=} contains deprecated {Task.Fine=}')
-
     if Task.FineTerrain in task:
         terrain.fine_terrain(output_folder)
 
@@ -305,17 +307,15 @@ def execute_tasks(
             for mesh in os.listdir(input_folder):
                 if (mesh.endswith(".glb") or mesh.endswith(".b_displacement.npy")) and not os.path.islink(output_folder / mesh):
                     os.symlink(input_folder / mesh, output_folder / mesh)
-        if Task.Coarse in task or Task.Populate in task or Task.FineTerrain in task:
-            bpy.context.preferences.system.scrollback = 100 
-            bpy.context.preferences.edit.undo_steps = 100
+        if Task.Coarse in task or Task.Populate in task:
+
             with Timer(f'Writing output blendfile'):
                 logging.info(f'Writing output blendfile to {output_folder / output_blend_name}')
                 bpy.ops.wm.save_mainfile(filepath=str(output_folder / output_blend_name))
                 tag_system.save_tag(path=str(output_folder / "MaskTag.json"))
 
             with (output_folder/ "version.txt").open('w') as f:
-                scene_version = get_scene_tag('VERSION')
-                f.write(f"{scene_version}\n")
+                f.write(f"{VERSION}\n")
 
             with (output_folder/'polycounts.txt').open('w') as f:
                 save_polycounts(f)
@@ -361,39 +361,25 @@ def apply_scene_seed(args):
     np.random.seed(scene_seed)
     return scene_seed
 
-def apply_gin_configs(args, scene_seed, skip_unknown=False):
-
-    scene_types = [p.stem for p in Path('config/scene_types').iterdir()]
-    scene_specified = any(s in scene_types or s.startswith("figure") for s in args.gin_config)
-
-    weights = {
-        "kelp_forest": 0.3,
-        "coral_reef": 1,
-        "forest": 2,
-        "river": 2,
-        "desert": 1,
-        "coast": 1,
-        "cave": 1,
-        "mountain": 1,
-        "canyon": 1,
-        "plain": 1,
-        "cliff": 1,
-        "arctic": 1,
-        "snowy_mountain": 1,
-    }
-    assert all(k in scene_types for k in weights)
-
-    scene_types = [s for s in scene_types if s in weights]
-    weights = np.array([weights[k] for k in scene_types], dtype=float)
-    weights /= weights.sum()
+@gin.configurable
+def apply_gin_configs(
+    args, 
+    scene_seed, 
+    skip_unknown=False, 
+    mandatory_config_dir=Path('config/scene_types'),
+):
 
-    if not scene_specified:
-        scene_type = np.random.RandomState(scene_seed).choice(scene_types, p=weights)
-        logging.warning(f'Randomly selected {scene_type=}. IF THIS IS NOT INTENDED THEN YOU ARE MISSING SCENE CONFIGS')
-        if len(args.gin_config) > 0 and args.gin_config[0] == 'base':
-            args.gin_config = [scene_type] + args.gin_config[1:]
-        else:
-            args.gin_config = [scene_type] + args.gin_config
+    if mandatory_config_dir is not None:
+        assert mandatory_config_dir.exists()
+        scene_types = [p.stem for p in mandatory_config_dir.iterdir()]
+        scenetype_specified = any(s in scene_types or s.split('.')[0] in scene_types for s in args.configs)
+
+        if not scenetype_specified:
+            print(scene_types)
+            raise ValueError(
+                f"Please load one or more config from {mandatory_config_dir} using --configs to avoid unexpected behavior. "
+                "If you are sure you want to proceed without, override `apply_gin_configs.mandatory_config_dir=None`"
+            )
 
     def find_config(g):
         for p in Path('config').glob('**/*.gin'):
@@ -403,8 +389,8 @@ def find_config(g):
                 return p
         raise ValueError(f'Couldn not locate {g} or {g}.gin in anywhere config/**')
 
-    bindings = sanitize_gin_override(args.gin_param)
-    confs = [find_config(g) for g in ['base.gin'] + args.gin_config]
+    bindings = sanitize_gin_override(args.overrides)
+    confs = [find_config(g) for g in ['base.gin'] + args.configs]
     gin.parse_config_files_and_bindings(confs, bindings=bindings, skip_unknown=skip_unknown)
 
 def main(

worldgen/generate.py

@@ -375,10 +375,10 @@ def main():
     parser.add_argument('-s', '--seed', default=None, help="The seed used to generate the scene")
     parser.add_argument('-t', '--task', nargs='+', default=['coarse'],
                         choices=['coarse', 'populate', 'fine_terrain', 'ground_truth', 'render', 'mesh_save'])
-    parser.add_argument('-g', '--gin_config', nargs='+', default=['base'],
+    parser.add_argument('-g', '--configs', nargs='+', default=['base'],
                         help='Set of config files for gin (separated by spaces) '
                              'e.g. --gin_config file1 file2 (exclude .gin from path)')
-    parser.add_argument('-p', '--gin_param', nargs='+', default=[],
+    parser.add_argument('-p', '--overrides', nargs='+', default=[],
                         help='Parameter settings that override config defaults '
                              'e.g. --gin_param module_1.a=2 module_2.b=3')
     parser.add_argument('--task_uniqname', type=str, default=None)