diff --git a/README.md b/README.md index 89bbf0ec1..a768b113a 100644 --- a/README.md +++ b/README.md @@ -2,7 +2,7 @@ Please visit our website, [https://infinigen.org](https://infinigen.org) -[![Infinigen Trailer](images/video_thumbnail.png)](https://youtu.be/6tgspeI-GHY) +[![Infinigen Intro Video](docs/images/video_thumbnail.png)](https://youtu.be/6tgspeI-GHY) If you use Infinigen in your work, please cite our [academic paper]([https://arxiv.org/abs/2306.09310](https://arxiv.org/abs/2306.09310)): @@ -24,166 +24,29 @@ Conference on Computer Vision and Pattern Recognition (CVPR) 2023 } ``` -## Installation +### Getting Started -Installation is tested and working on the following platforms: +First, follow our [Installation Instructions](docs/Installation.md). -- Ubuntu 22.04.2 LTS - - GPUs options tested: CPU only, GTX-1080, RTX-2080, RTX-3090, RTX-4090 (Driver 525, CUDA 12.0) - - RAM: 16GB -- MacOS Monterey & Ventura, Apple M1 Pro, 16GB RAM - -We are working on support for rendering with AMD GPUs. Windows users should use [WSL2](https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-11-with-gui-support#1-overview). More instructions coming soon. - -
-:warning: Errors with git pull / merge conflicts when migrating from v1.0.0 to v1.0.1 -To properly display open-source line by line git credits for our team, we have switched to a new version of the repo which does not share commit history with the the version available from 6/17/2023 to 6/29/2023 date. We hope this will help open source contributors identify the current "code owner" or person best equipped to support you with issues you encounter with any particular lines of the codebase. - -You will not be able to pull or merge infinigen v1.0.1 into a v1.0.0 repo without significant git expertise. If you have no ongoing changes, we recommend you clone a new copy of the repo. We apologize for any inconvenience, please make an issue if you have problems updating or need help migrating ongoing changes. We understand this change is disruptive, but it is one-time-only and will not occur in future versions. Now it is complete, we intend to iterate rapidly in the coming weeks, please see our [roadmap](https://infinigen.org/roadmap) and [twitter](https://twitter.com/PrincetonVL) for updates. -
- -**Run these commands to get started** -``` -git clone --recursive https://github.com/princeton-vl/infinigen.git -cd infinigen -conda create --name infinigen python=3.10 -conda activate infinigen -bash install.sh -``` -`install.sh` may take significant time to download Blender3.3 and compile all source files. - -Ignore non-fatal warnings. See [Getting Help](#getting-help) for guidelines on posting github issues - -Run the following or add it to your `~/.bashrc` (Linux/WSL) or `~/.bash_profile` (Mac) -``` -# on Linux/WSL -export BLENDER="/PATH/TO/infinigen/blender/blender" -# on MAC -export BLENDER="/PATH/TO/infinigen/Blender.app/Contents/MacOS/Blender" -``` - -
-(Optional) Running Infinigen in a Docker Container - -**Docker on Linux** - -In `/infinigen/` -``` -make docker-build -make docker-setup -make docker-run -``` -To enable CUDA compilation, use `make docker-build-cuda` instead of `make docker-build` - -To run without GPU passthrough use `make docker-run-no-gpu` - -To run without OpenGL ground truth use `docker-run-no-opengl` - -To run without either, use `docker-run-no-gpu-opengl` - -Note: `make docker-setup` can be skipped if not using OpenGL. - -Use `exit` to exit the container and `docker exec -it infinigen bash` to re-enter the container as needed. Remember to `conda activate infinigen` before running scenes. - - -**Docker on Windows** - -Install [WSL2](https://infinigen.org/docs/installation/intro#setup-for-windows) and [Docker Desktop](https://www.docker.com/products/docker-desktop/), with "Use the WSL 2 based engine..." enabled in settings. Keep the Docker Desktop application open while running containers. Then follow instructions as above. - -
- -### "Hello World": Generate your first Infinigen scene - - :warning: **Known issue** : We are actively fixing an issue which causes commands not to be reproducible on many platforms. The same command may produce multiple rearranged scenes with different runtimes and memory requirements. +Next, see our ["Hello World" example](docs/HelloWorld.md) to generate an image & ground truth similar to those shown below.

- - - - + + + +

-This guide will show you how to generate an image and it's corresponding ground-truth, similar to those shown above. - -#### Generate a scene step by step -Infinigen generates scenes by running multiple tasks (usually executed automatically, like in [Generate image(s) in one command](#generate-images-in-one-command)). Here we will run them one by one to demonstrate. These commands take approximately 10 minutes and 16GB of memory to execute on an M1 Mac or Linux Desktop. - -``` -cd worldgen -mkdir outputs - -# Generate a scene layout -$BLENDER -noaudio --background --python generate.py -- --seed 0 --task coarse -g desert simple --output_folder outputs/helloworld/coarse - -# Populate unique assets -$BLENDER -noaudio --background --python generate.py -- --seed 0 --task populate fine_terrain -g desert simple --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine +### Documentation -# Render RGB images -$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames +- [Installation Guide](docs/Installation.md) +- ["Hello World": Generate your first Infinigen scene](docs/HelloWorld.md) +- [Configuring Infinigen](docs/ConfiguringInfinigen.md) +- [Extended ground-truth](docs/GroundTruthAnnotations.md) +- [Generating individual assets](docs/GeneratingIndividualAssets.md) -# Render again for accurate ground-truth -$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert simple --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image -``` - -Output logs should indicate what the code is working on. Use `--debug` for even more detail. After each command completes you can inspect it's `--output_folder` for results, including running `$BLENDER outputs/helloworld/coarse/scene.blend` or similar to view blender files. We hide many meshes by default for viewport stability; to view them, click "Render" or use the UI to unhide them. - -#### [Extended ground-truth & docmentation](./GroundTruthAnnotations.md) - -We also provide a (optional) separate pipeline for extracting the full set of annotations from each image or scene. Refer to [GroundTruthAnnotations.md](./GroundTruthAnnotations.md) for compilation instructions, data format specifications and an extended "Hello World". - -#### Generate image(s) in one command - -We provide `tools/manage_datagen_jobs.py`, a utility which runs these or similar steps automatically. - -``` -python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 ---configs desert simple --pipeline_configs local_16GB monocular blender_gt --pipeline_overrides LocalScheduleHandler.use_gpu=False -``` - -Ready to remove the guardrails? Try the following: -- Swap `desert` for any of `config/scene_types` to get different biome (or write your own crazy config!). You can also add in the name of any file in `configs`. -- Change the `--specific_seed` to any number to produce different scenes, or remove it and set --num_scenes 50 to try many random seeds. -- Remove `simple` to generate more detailed (but *EXPENSIVE*) scenes, as shown in the trailer. -- Read and customize `generate.py` to understand how infinigen works under the hood. -- Append `-p compose_scene.grass_chance=1.0` to the first command to force grass (or any of `generate.py's` 'run_stage' asset names) to appear in the scene. You can modify the kwargs @gin.configurable() python function in the entire repo via this mechanism. - -`--configs` enables you to customize the random *distribution* of visual content. If you do not select any config in the folder `config/scene_types`, the code choose one for you at random. - -`--pipeline_configs` determines what compute resources will be used, and what render jobs are necessary for each scene. A list of configs are available in `tools/pipeline_configs`. You must pick one config to determine compute type (ie `local_64GB` or `slurm`) and one to determine the dataset type (such as `monocular` or `monocular_video`). Run `python -m tools.manage_datagen_jobs --help` for more options related to dataset generation. - -If you intend to use CUDA-accelerated terrain (`--pipeline_configs cuda_terrain`), you must run `install.sh` on a CUDA-enabled machine. - -Infinigen uses [Google's "Gin Config"](https://github.com/google/gin-config) heavily, and we encourage you to consult their documentation to familiarize yourself with its capabilities. - -
-:bulb: Generating high quality videos / Avoiding terrain aliasing - -To render high quality videos as shown in the intro video, we ran commands similar to the following, on our SLURM cluster. - -``` -python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ - --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ - --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain -``` - -Our terrain system resolves its signed distance function (SDF) to view-specific meshes, which must be updated as the camera moves. For video rendering, we strongly recommend using the `high_quality_terrain` config to avoid perceptible flickering and temporal aliasing. This config meshes the SDF at very high detail, to create seamless video. However, it has high compute costs, so we recommend also using `--pipeline_config cuda_terrain` on a machine with an NVIDIA GPU. For applications with fast moving cameras, you may need to update the terrain mesh more frequently by decreasing `iterate_scene_tasks.view_block_size = 16` in `worldgen/tools/pipeline_configs/monocular_video.gin` - -As always, you may attempt to switch the compute platform (e.g from `slurm` to `local_256GB`) or the data format (e.g. from `monocular_video` to `stereo_video`). - -
- -#### Generating and using ground-truth - -:exclamation: Infinigen provides a ground-truth system for generating diverse automatic annotations for computer vision. [See the docs here](/GroundTruthAnnotations.md). - -## Exploring the Infinigen Codebase - -Infinigen has evolved significantly since the version described in our CVPR paper. It now features some procedural code obtained from the internet under CC-0 licenses, which are marked with code comments where applicable - no such code was present in the system for the CVPR version. - -Infinigen is an ongoing research project, and has some known issues. Through experimenting with Infinigen's code and config files, you will find scenes which crash or cannot be handled on your hardware. Infinigen scenes are randomized, with a long tail of possible scene complexity and thus compute requirements. If you encounter a scene that does not fit your computing hardware, you should try other seeds, use other config files, or follow up for help. - -## Coming Soon -Please see our project roadmap and follow us at [https://twitter.com/PrincetonVL](https://twitter.com/PrincetonVL) for updates. +### Coming Soon +Please see our [project roadmap](https://infinigen.org/roadmap) and follow us at [https://twitter.com/PrincetonVL](https://twitter.com/PrincetonVL) for updates. ### Contributing We welcome contributions! You can contribute in many ways: @@ -191,7 +54,7 @@ We welcome contributions! You can contribute in many ways: - **Contribute procedural generators** - `worldgen/nodes/node_transpiler/dev_script.py` provides tools to convert artist-friendly [Blender Nodes](https://docs.blender.org/manual/en/2.79/render/blender_render/materials/nodes/introduction.html) into python code. Tutorials and guidelines coming soon. - **Contribute pre-generated data** - Anyone can contribute their computing power to create data and share it with the community. Please stay tuned for a repository of pre-generated data. -## Getting Help +### Getting Help Please post this repository's Github Issues page for help. Please run your command with `--debug`, and let us know: - What is your computing setup, including OS version, CPU, RAM, GPU(s) and any drivers? @@ -201,7 +64,7 @@ Please post this repository's Github Issues page for help. Please run your comma - If using `manage_datagen_jobs`, look in `outputs/MYJOB/MYSEED/logs/` to find the right one. - What was the exact python error and stacktrace, if applicable? -## Acknowledgements +### Acknowledgements Infinigen wouldn't be possible without the fantastic work of the [Blender Foundation](https://www.blender.org/) and it's open-source contributors. Infinigen uses many open source projects, with special thanks to [Land-Lab](https://github.com/landlab/landlab), [BlenderProc](https://github.com/DLR-RM/BlenderProc) and [Blender-Differential-Growth](https://github.com/inca/blender-differential-growth). @@ -232,4 +95,6 @@ We learned tremendously from online tutorials of [Ryan King Art](https://www.youtube.com/@RyanKingArt), [Sam Bowman](https://www.youtube.com/@snow_mamba) and [yogigraphics](https://www.youtube.com/@yojigraphics). -These tutorials provided procedural generators for our early experimentation and served as inspiration for our own implementations in the official release of Infinigen. They are acknowledged in file header comments where applicable. +These tutorials provided procedural generators for our early experimentation and served as inspiration for our own implementations in the official release of Infinigen. They are acknowledged in file header comments where applicable. + +Infinigen has evolved significantly since the version described in our CVPR paper. It now features some procedural code obtained from the internet under CC-0 licenses, which are marked with code comments where applicable - no such code was present in the system for the CVPR version. diff --git a/CHANGELOG.md b/docs/CHANGELOG.md similarity index 100% rename from CHANGELOG.md rename to docs/CHANGELOG.md diff --git a/docs/ConfiguringInfinigen.md b/docs/ConfiguringInfinigen.md new file mode 100644 index 000000000..d1fa2124b --- /dev/null +++ b/docs/ConfiguringInfinigen.md @@ -0,0 +1,237 @@ +# Configuring Infinigen + +This document explains how to configure various features of Infinigen. It assumes you have already completed our Installation instructions and [Hello World](HelloWorld.md) example. + +:exclamation: By experimenting with Infinigen's code and config files, you will find scenes which crash or cannot be handled on your hardware. Infinigen scenes are randomized, with a long tail of possible scene complexity and thus compute requirements. If you encounter a scene that does not fit your computing hardware, you should try other seeds, use other config files, or follow up for help. + +## Overview + +Generating scenes with Infinigen typically involves two main python scripts: +1. [generate.py](../worldgen/generate.py) - our example scene composition script, which invokes and places assets to create a realistic nature scene. +1. [manage_datagen_jobs.py](../worldgen/tools/manage_datagen_jobs.py) - a script which invokes the above scene composition script many times to generate a useful dataset. + +`manage_datagen_jobs.py` controls how many and what jobs will be run, and `generate.py` determines what will happen during those jobs. Ultimately both programs must be configured correctly in order to make useful data. + +### Scene tasks + +To complete one Infinigen scene, `manage_datagen_jobs.py` will run several "tasks", each composed of a single execution of `generate.py`(You ran similar processes yourself in the step by step part of "Hello World"). + +Typically, these tasks are as follows: + 1. `coarse`, which generates coarse terrain shape, puts placeholders for creatures and large trees / obstacles, and generates all small assets (twigs, particles, etc) that will be "instanced", IE scattered across the terrain with repetition. + 1. `populate`, which replaces the placeholders from the previous step with unique, detailed assets (trees, creatures, etc) provided they are necessary in the final camera trajectory. + 1. `rendershort` - render the scene using Blender's CYCLES renderer. If working on a video, this step renders a short block (usually 16 frames) of adjacent video. + 1. `renderbackup` - retry the above step with more RAM etc, to make sure high-cost, detailed scenes (which can crash in the previous step) still complete. + 1. (if enabled via `--pipeline_configs`)`blender_gt` or `opengl_gt` - extract ground truth labels for AI training etc. + +`coarse` and `populate` typically run once per scene, whereas `fine_terrain` as well as any render or ground truth tasks may run many times for different cameras or video timesteps. + +The outputs for all tasks belonging to a particular scene will be stored in the `outputs/MYJOB/SEED`, where MYJOB was the folder name chosen in the `--output_folder` argument, and `SEED` is the random seed used to create the scene. Each task creates its own detailed log files in `outputs/MYJOB/SEED/logs/TASKNAME.log` and `outputs/MYJOB/SEED/logs/TASKNAME.err` - please check these logs to see detailed error messages if your jobs crash . + +Infinigen is designed to run many independent scenes in paralell. This means that even if running all the steps for each scene can take some time, we can still achieve good throughput and utilization (provided it has enough resources - most laptops only have enough RAM for one scene at a time). + +#### Overrides and Config Files + +Both `manage_datagen_jobs.py` and `generate.py` can be configured via the commandline or config files, using [Google's "Gin Config"](https://github.com/google/gin-config). Gin allows you to insert new default keyword arguments ("kwargs") for any function decorated with `@gin.configurable`; many such functions exist in our codebase, and via gin overrides you can create datsets suiting many diverse applications, as is explained in the coming sections. + +To use gin, simply add commandline arguments such as `-p compose_scene.rain_particles_chance = 1.0` to override the chance of rain, or `--pipeline_overrides iterate_scene_tasks.frame_range=[1,25]` to set a video's length to 24 frames. You can chain many statements together, separated by spaces, to configure many parts of the system at once. These statements depend on knowing the python names of the function and keyword argument you wish to override. To find parameters you wish to override, you should browse `worldgen/config/base.gin` and other configs, or `generate.py` and the definitions of any functions it calls. Better documentation and organization of the available parameters will come in future versions. + +If you find a useful and related combination of these commandline overrides, you can write them into a `.gin` file in `worldgen/configs`. Then, to load that config, just include the name of the file into the `--configs`. If your overrides target `manage_datagen_jobs` rather than `generate.py` you should place the config file in `worldgen/tools/pipeline_configs` and use `--pipeline_configs` rather than `--configs`. + +Our `generate.py` driver always loads [`worldgen/configs/base.gin`][../worldgen/configs/base.gin], and you can inspect / modify this file to see many common and useful gin override options. + +`generate.py` also expects that one file from (configs/scene_types/)[worldgen/config/scene_types] will be loaded, and if one is not specified on the commandline it will choose one randomly according to the keys and weights in `worldgen/core.py`. These scene_type configs contain gin overrides designed to encode the semantic constraints of real natural habitats (e.g. `worldgen/scene_types/desert.gin` causes sand to appear and cacti to be more likely). + +### Moving beyond "Hello World" + +Now that you understand the two major python programs and how to configure them, you may notice and wonder about the many configs/overrides provided in our original one-command "Hello World" example: + +``` +# Original hello world command +python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 \ +--configs desert.gin simple.gin --pipeline_configs local_16GB.gin monocular.gin blender_gt.gin \ +--pipeline_overrides LocalScheduleHandler.use_gpu=False +``` + +Here is a breakdown of what every commandline argument does, and ideas for how you could change them / swap them out: + - `--output_folder outputs/hello_world` - change this to change where the files end up + - `--specific_seed 0` forces the system to use a random seed of your choice, rather than choosing one at random. Change this seed to get a different random variation, or remove it to have the program choose a seed at random + - `--num_scenes` decides how many unique scenes the program will attempt to generate before terminating. Once you have removed `--specific_seed`, you can increase this to generate many scenes in sequence or in paralell. + - `--configs desert.gin simple.gin` forces the command to generate a desert scene, and to do so with relatively low mesh detail, low render resolution, low render samples, and some asset types disabled. + - Do `--configs snowy_mountain.gin simple.gin` to try out a different scene type (`snowy_mountain.gin` can instead be any scene_type option from `worldgen/configs/scene_types/`) + - Remove the `desert.gin` and just specify `--configs simple.gin` to use random scene types according to the weighted list in `worldgen/core.py`. + - You have the option of removing `simple.gin` and specify neither of the original configs. This turns off the many detail-reduction options included in `simple.gin`, and will create scenes closer to those in our intro video, albeit at significant compute costs. Removing `simple.gin` will likely cause crashes unless using a workstation/server with large amounts of RAM and VRAM. You can find more details on optimizing scene content for performance [here](#config-overrides-for-mesh-detail-and-performance). + - `--pipeline_configs local_16GB.gin monocular.gin blender_gt.gin` + - `local_16GB.gin` specifies to run only a single scene at a time, and to run each task as a local python process. See [here](#configuring-available-computing-resources) for more options + - `monocular.gin` specifies that we want a single image per scene, not stereo or video. See [here](#rendering-video-stereo-and-other-data-formats) for more options. + - `blender_gt.gin` specifies to extract ground truth labels (depth, surface normals, etc) using Blender's in-built render. If you do not need these, remove this config to save on runtime. + - `--pipeline_overrides LocalScheduleHandler.use_gpu=False` tells the system not to look for available GPUs, and to _not_ make them available to any jobs. This is intended only to make the Hello World easier to run, and work on non-NVIDIA systems. Please [click here](#using-gpu-acceleration) for full instructions on using a GPU. + +## Commandline Options in Detail + +This section highlights commandline options to help you deploy Infinigen on your compute, enable GPU usage, render various datasets, tune resolution/compute costs, and more. + +[Click here to skip to Example Commands](#example-commands) + +### Configuring available computing resources + +`worldgen/tools/pipeline_configs/compute_platform` provides several preset configs to help you run the appropriate number and type of Infinigen tasks suitable to your hardware.
These configs are mutually exclusive, but you must include at least one.
+ +`local_16GB.gin` through `local_256GB.gin` are intended for laptops, desktop workstations, or a single headless node. These configs run each task as a child process on the same machine that ran `manage_datagen_jobs.py`. Each config runs the right number of concurrent jobs for the specified amount of RAM. If you have a different amount of RAM than the options provided, you should override `manage_datagen_jobs.num_concurrent`to your amount of system RAM divided by 20GB. Ultimately, the amount of concurrent process you can run will depend on the format (single-image vs. video) and complexity (detail, clutter, etc) of the scenes you wish to generate - see [here](#config-overrides-for-mesh-detail-and-performance) to customize. + +`slurm.gin` is a special config designed to deploy Infinigen onto SLURM computing clusters. When using this config, each task will be executed as a remote slurm job (using [submitit](https://github.com/facebookincubator/submitit)), with time-limit/CPU/RAM requests set as specified in `slurm.gin`. If your group has a SLURM partition name, please set `PARTITION = "mygroupname"` in `slurm.gin`. If you are able to run very long SLURM jobs (>= ~3 days) you may also consider submitting `manage_datagen_jobs` with `local_256GB.gin` as a single long slurm job. + +Please submit a Github Issue for help with deploying Infinigen on your compute. + +### Using GPU acceleration + +Infinigen currently only supports NVIDIA GPUs. Infinigen can use a GPU in accelerate the following tasks: + 1. In the `rendershort`, `renderbackup`, `blender_gt` tasks, Blender's CYCLES rendering engine can use a GPU to massively accelerate ray-tracing. This all-but-essential for video rendering, which require a large amount of render jobs. AMD/etc support for this step is possible but not currently supported/implemented. + 1. In the `fine_terrain` step, our custom marching cubes terrain generator has been specialized to run with CUDA, to massively improve runtime and allow higher detail terrain. + 1. In the `opengl_gt` step (if enabled) our custom ground truth code uses OpenGL and thus requires access to a GPU. + +To enable these GPU accelerated steps: + - First, if you are using a `local_*.gin` pipeline config, you must first remove `--pipeline_overrides LocalScheduleHandler.use_gpu=False` from our Hello World command, or otherwise ensure this value is set to true via configs/overrides. This will make the GPU _visible_ to each child process, and will cause _rendering_ to automatically detect and use the GPU. `slurm.gin` assumes GPUs will be available by default, set it's GPU request amounts to 0 if this is not the case for your cluster. + - To enable GPU-acceleration for `fine_terrain`, you must ensure that `install.sh` was run on a machine with CUDA, then add `cuda_terrain.gin` to your `--pipeline_configs`. + - OpenGL GT can be enabled described in [Extended ground-truth](GroundTruthAnnotations.md) + +Even if GPU access is enabled, *you will likely not see the GPU at 100% usage*. Both blender and our code require some CPU setup time to build acceleration data-structures before GPU usage can start, and even then, a single render job may not saturate your GPU's FLOPS. + - If you are using a GPU with >=32 GB VRAM, you may consider `--pipeline_override LocalScheduleHandler.jobs_per_gpu=2` to put 2 render jobs on each GPU and increase utilization. + +If you have more than one GPU and are using a `local_*.gin` compute config, each GPU-enabled task will be assigned to a single GPU, and you will need to run many concurrent jobs to keep your GPUs fed. Blender natively supports multi-GPU rendering, but we do not use this by default - create a Github Issue \[REQUEST\] if your application requires this. + +### Rendering Video, Stereo and other data formats + +Generating a video, stereo or other dataset typically requires more render jobs, so we must instruct `manage_datagen_jobs.py` to run those jobs. `worldgen/tools/pipeline_configs/data_schema/` provides many options for you to use in your `--pipeline_configs`, including `monocular_video.gin` and `stereo.gin`.
These configs are typically mutually exclusive, and you must include at least one
+ +:exclamation: Our terrain system resolves its signed distance function (SDF) to view-specific meshes, which must be updated as the camera moves. For video rendering, we strongly recommend using the `high_quality_terrain` config to avoid perceptible flickering and temporal aliasing. This config meshes the SDF at very high detail, to create seamless video. However, it has high compute costs, so we recommend also using `--pipeline_config cuda_terrain` on a machine with an NVIDIA GPU. For applications with fast moving cameras, you may need to update the terrain mesh more frequently by decreasing `iterate_scene_tasks.view_block_size = 16`. + +To create longer videos, modify `iterate_scene_tasks.frame_range` in `monocular_video.gin` (note: we use 24fps video by default). `iterate_scene_tasks.view_block_size` controls how many frames will be grouped into each `fine_terrain` and render / ground-truth task. + +If you need more than two cameras, or want to customize their placement, see `worldgen/config/base.gin`'s `camera.spawn_camera_rigs.camera_rig_config` for advice on existing options, or write your own code to instantiate a custom camera setup. + +### Config Overrides to Customize Scene Content + +:bulb: If you only care about few specific assets, or want to export Infinigen assets to another project, instead see [Generating individual assets](GeneratingIndividualAssets.md). + +You can achieve a great deal of customization by browsing and editing `worldgen/config/base.gin` - e.g. modifying cameras, lighting, asset placement, etc. + - `base.gin` only provides the default values of these configs, and may be overridden by scene_type configs. To apply a setting globally across all scene types, you should put them in a new config placed at the end of your `--configs` argument (so that it's overrides are applied last), or use commandline overrides. + +However, many options exist which are not present in base.gin. At present, you must browse `generate.py` to find the part of the code you wish to customize, and look through the relevant code for what more advanced @gin.configurable functions are available. You can also add @gin.configurable to most functions to allow additional configuration. More documentation on available parameters is coming soon. + +For most steps of `generate.py`'s `compose_scene` function, we use our `RandomStageExecutor` wrapper to decide whether the stage is run, and handle other bookkeeping. This means that if you want to decide the probability with which some asset is included in a scene, you can use the gin override `compose_scene.trees_chance=1.0` or something similar depending on the name string provided as the first argument of the relevant run_stage calls in this way, e.g. `compose_scene.rain_particles_chance=0.9`to make most scenes rainy, or `compose_scene.flowers_chance=0.1` to make flowers rarer. + +A common request is to just turn off things you don't want to see, which can be achieved by adding `compose_scene.trees_chance=0.0` or similar to your `-p` argument or a loaded config file. To conveniently turn off lots of things at the same time, we provide configs in `worldgen/configs/disable_assets` to disable things like all creatures, or all particles. + +You will also encounter configs using what we term a "registry pattern", e.g. `worldgen/config/base_surface_registry.gin`'s `ground_collection`. "Registries", in this project, are a list of discrete generators, with weights indicating how relatively likely they are to be chosen each time the registry is sampled. + - For example, in `base_surface_registry.gin`, `surface.registry.beach` specifies `("sand", 10)` to indicate that sand has high weight to be chosen to be assigned for the beach category. + - Weights are normalized by their overall sum to obtain a probability distribution. + - Name strings undergo lookup in the relevant source code folders, e.g. the name "sand" in a surface registry maps to `worldgen/surfaces/templates/sand.py`. + - The random choice among scene_type configs is itself a registry, although it is hardcoded in `core.py` currently, since the choice of what configs are loaded cannot depend on a config file. This will be improved soon. + +### Config Overrides for mesh detail and performance + +The quantity, diversity and detail of assets in a scene drastically affects RAM/VRAM requirements and runtime. This section will highlight configurable parameters that may help tune Infinigen to run better on limited hardware, or that could be increased to create larger more detailed scenes. + +Infinigen can generate meshes at a wide range of resolutions. Many gin-configurable options exist to customize how detailed our meshes will be.
These options, as well as the choice of scene type configs, are the most effective ways to decrease compute costs.
+- All mesh resolutions are defined in terms of pixels-per-face. Increasing/decreasing the `compose_scene.generate_resolution` will have corresponding effects on geometry detail. If you wish to render the same mesh at a different resolution, override `render_image.render_resolution_override` +- `OpaqueSphericalMesher.pixels_per_cube` and `TransparentSphericalMesher.pixels_per_cube`. You can increase them from their default, 2 pixels per marching cube, to 4 (as seen in `dev.gin`) or higher, in order to reduce terrain resolution and cost. Low resolution terrain will cause noticeable artifacts in videos, you must use `high_quality_terrain.gin` if rendering videos with moving cameras. +- `target_face_size.global_multiplier` controls the resolution of all other assets. Increase it to 4 or higher to reduce the compute cost of non-terrain assets like plants and creatures. +- All of these options have diminishing returns - a minimal amount of geometry or data is always generated to help define the shape of each asset, which may be larger than the final geometry if you set resolution very low. + +Infinigen curbs memory costs by only populating assets up to a certain distance away from the camera (except for terrain, which is essentially unbounded). `base.gin` contains many options to customize how far away, and thus how many, assets will be placed: + - `placement.populate_all.dist_cull` controls the maximum distance to populate placeholders (for assets such as trees, cacti, creatures, etc). Reducing this will curb the number of assets and especially the number of trees, which can be quite expensive. + - `compose_scene.inview_distance`control the maximum distance to scatter plants/rocks/medium-sized-objects. Reducing this will reduce the number of poses for these assets, but will not reduce the number of unique underlying meshes, so may have diminishing returns. + - Similarly to the above, `compose_scene.near_distance` controls the maximum distance to scatter tiny particles like pine needles. + - Infinigen does not populate assets which are far outside the camera frustrum. You may attempt to reduce camera FOV to minimize how many assets are in view, but be warned there will be minimal or significantly diminishing returns on performance, due to the need to keep out-of-view assets loaded to retain accurate lighting/shadows. + +We also provide `worldgen/configs/performance/dev.gin`, a config which sets many of the above performance parameters to achieve lower scenes. We often use this config to obtain previews for development purposes, but it may also be suitable for generating lower resolution images/scenes for some tasks. + +Our current system determines asset mesh resolution based on the _closest distance_ it comes to the camera during an entire trajectory. Therefore, longer videos are more expensive, as more assets will be closer to the camera at some point in the trajectory. Options exist to re-generate assets at new resolutions over the course of a video to curb these costs - please make a Github Issue for advice. + +If you find yourself bottlenecked by GPU time, you should consider the following options: + - Render single images or stereo images, rather than video, such that less render jobs are required for each CPU-bound `coarse`/`populate` job + - Reduce `base.gin`'s `full/render_image.num_samples = 8192` or `compose_scene.generate_resolution = (1920, 1080)`. This proportionally reduces rendering FLOPS, with some diminishing returns due to BVH setup time. + - If your GPU(s) are _underutilized_, try the reverse of these tips. + +Some scene type configs are also generally more expensive than others. `forest.gin` and `coral.gin` are very expensive due to dense detailed fauna, wheras `artic` and `snowy_mountain` are very cheap. Low-resource compute settings (<64GB) of RAM may only be able to handle a subset of our `worldgen/config/scene_type/` options, and you may wish to tune the ratios of scene_types by editing `worldgen/core.py`. + +### Other `manage_datagen_jobs.py` commandline options + +Please run `pythom -m tools.manage_datagen_jobs --help` for an up-to-date description of other commandline arguments. We always use `--cleanup big_files --warmup_sec 30000` for large render jobs. Optionally, you can also log render progress to Weights & Biases. + +## Example Commands + +Below we provide many example commands we have found useful to generate scenes/images/videos. + +If you have any issues with these commands, or wish to customize them to your needs,
we encourage you to read the above documentation
, then follow up on our Github Issues for help. + +All commands below are shown with using `local_256GB` config, but you can attempt to swap this for any compute config as discussed in [Configuring available computing resources](#configuring-available-computing-resources). + +#### Creating videos similar to the intro video + +Most videos in the "Introducing Infinigen" launch video were made using commands similar to the following: + +```` +python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ + --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain +```` + +#### Creating large-scale stereo datasets + +```` +python -m tools.manage_datagen_jobs --output_folder outputs/stereo_data --num_scenes 10000 \ + --pipeline_config slurm stereo cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain +```` + +#### Creating a few low-resolution images to your test changes + +``` +screen python -m tools.manage_datagen_jobs --output_folder outputs/dev --num_scenes 50 \ + --pipeline_config slurm monocular cuda_terrain \ + --cleanup big_files --warmup_sec 1200 --configs dev +``` + +#### Creating datasets with specific properties or constraints: + +These commands are intended as inspiration - please read docs above for more advice on customizing all aspects of Infinigen. + + Create images that always have rain: +``` +python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ + --pipeline_config slurm monocular cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 30000 \ + --overrides compose_scene.rain_particles_chance=1.0 +``` + +:bulb: You can substitute the `rain_particles` in `rain_particles_chance` for any `run_stage` name argument string in `worldgen/generate.py`, such as `trees` or `ground_creatures`. + + Create images that only have terrain: +``` +python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ + --pipeline_config slurm monocular cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 30000 --config no_assets +``` +:bulb: You can substitute "no_assets" for `no_creatures` or `no_particles`, or the name of any file under `worldgen/config`. The command shown uses `worldgen/config/disable_assets/no_assets.gin`. + + Create videos at birds-eye-view camera altitudes: + +``` +python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ + --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 30000 --config trailer high_quality_terrain \ + --overrides camera.camera_pose_proposal.altitude=["uniform", 20, 30] +``` + +:bulb: The command shown is overriding `worldgen/config/base.gin`'s default setting of `camera.camera_pose_proposal.altitude`. You can use a similar syntax to override any number of .gin config entries. Separate multiple entries with spaces. + + Create 1 second video clips: +``` +python -m tools.manage_datagen_jobs --output_folder outputs/my_videos --num_scenes 500 \ + --pipeline_config slurm monocular_video cuda_terrain opengl_gt \ + --cleanup big_files --warmup_sec 30000 --config trailer high_quality_terrain \ + --pipeline_overrides iterate_scene_tasks.frame_range=[1,25] +``` + +:bulb: This command uses `--pipeline_overrides` rather than `--overrides` since it is providing a gin override to the `manage_datagen_jobs.py` process, not some part main `generate.py` driver. diff --git a/docs/GeneratingIndividualAssets.md b/docs/GeneratingIndividualAssets.md new file mode 100644 index 000000000..774f6a838 --- /dev/null +++ b/docs/GeneratingIndividualAssets.md @@ -0,0 +1,38 @@ +# Generating Individual Assets + +This tutorial will help you generate images or .blend files of specific assets of your choice. + +Limitations (to be addressed soon): +- This tool only exports .blend files. [See here](../worldgen/tools/export/README.md) for a prototype tool to convert these to standard mesh file formats, but it itself has some limitations. +- This tool cannot currently generate or export terrain meshes. + +### Example Commands + +Shown are three examples of using our `generate_individual_assets.py` script to create images and .blend files. + +```bash +cd worldgen +mkdir outputs +$BLENDER --background --python tools/generate_individual_assets.py -- -f CoralFactory -n 8 --save_blend +$BLENDER --background --python tools/generate_individual_assets.py -- -f seashells -n 1 --save_blend +$BLENDER --background --python tools/generate_individual_assets.py -- -f chunkyrock -n 1 --save_blend +``` + +

+ + + +

+ +Running the above commands will save images and .blend files into your `outputs` folder. + +Please run `$BLENDER --background --python tools/generate_individual_assets.py -- --help` for a full list of commandline arguments. + +The most commonly used arguments are: +- `-f` to specify the name(s) of assets or materials to generate. `-f NAME` can specify to generate three different types of objects: + - If `NAME` is the name of a class defined in `worldgen/assets`, then it will be treated as an AssetFactory and used to generate objects from scratch. For example, you can say `-f CactusFactory` or `-f CarnivoreFactory`, or use the name of any similar Factory class in the codebase. + - If `NAME` is the name of a file in `worldgen/surfaces/templates`, that material will be applied onto a sphere + - If `NAME` is the name of a file in `worldgen/surfaces/scatters`, that scatter generator will be applied nto a plane +- `-n` adjusts the number of images / blend files to be generated. + + diff --git a/GroundTruthAnnotations.md b/docs/GroundTruthAnnotations.md similarity index 95% rename from GroundTruthAnnotations.md rename to docs/GroundTruthAnnotations.md index e68a6de02..a458bb1c8 100644 --- a/GroundTruthAnnotations.md +++ b/docs/GroundTruthAnnotations.md @@ -122,7 +122,7 @@ Depth is stored as a 2160 x 3840 32-bit floating point numpy array. *Visualization:* `summary_json["Depth"]["png"]["00"]["00"]["0001"]` -> `frames/Depth_0001_00_00.png`

- +

The depth and camera parameters can be used to warp one image to another frame by running: @@ -141,7 +141,7 @@ The coordinate system for the surface normals is +X -> Right, +Y -> Up, +Z Backw *Visualization:* `summary_json["SurfaceNormal"]["png"]["00"]["00"]["0001"]` -> `frames/SurfaceNormal_0001_00_00.png`

- +

### Occlusion Boundaries :large_blue_diamond: @@ -151,7 +151,7 @@ Occlusion Boundaries are stored as a 2160 x 3840 png, with 255 indicating a boun *Path/Visualization:* `summary_json["OcclusionBoundaries"]["png"]["00"]["00"]["0001"]` -> `frames/OcclusionBoundaries_0001_00_00.png`

- +

### Optical Flow @@ -245,7 +245,7 @@ python tools/ground_truth/bounding_boxes_3d.py outputs/helloworld 1 --query blen which will output

- +

By ommitting the --query flag, a list of available tags will be printed. @@ -256,7 +256,7 @@ One could also produce a mask for only *flower petals*: python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query petal ```

- +

A benefit of our tagging system is that one can produce a segmentation mask for things which are not a distinct object, such as terrain attributes. For instance, we can highlight only *caves* or *warped rocks* @@ -266,7 +266,7 @@ python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query ca python tools/ground_truth/segmentation_lookup.py outputs/helloworld 1 --query warped_rocks ```

- +

### Object Metadata and 3D bounding boxes diff --git a/docs/HelloWorld.md b/docs/HelloWorld.md new file mode 100644 index 000000000..4c6e39b2f --- /dev/null +++ b/docs/HelloWorld.md @@ -0,0 +1,51 @@ +# "Hello World": Generate your first Infinigen scene + +

+ + + + +

+ +This guide will show you how to generate an image and it's corresponding ground-truth, similar to those shown above. + +:warning: **Known issue** : We are actively fixing an issue which causes Infinigen not to be reproducible on many platforms. The same command may produce multiple rearranged scenes with different runtimes and memory requirements, including those shown in "Hello World" and our [ConfiguringInfinigen.md](docs/ConfiguringInfinigen.md) + +## Generate a scene step by step +Infinigen generates scenes by running multiple tasks (usually executed automatically, like in [Generate image(s) in one command](#generate-images-in-one-command)). Here we will run them one by one to demonstrate. These commands take approximately 10 minutes and 16GB of memory to execute on an M1 Mac or Linux Desktop. + +:exclamation: If you encounter any missing .so files, missing dependencies (such as `gin`), or similar crashes, please check again that all steps of installation ran successfully. If you cannot resolve any issues with installation, please see our README and 'Bug Report' Git Issue template for advice on posting Git Issues to get help quickly - you must include the full installation logs in your issue so that we can help debug. + +``` +cd worldgen +mkdir outputs + +# Generate a scene layout +$BLENDER -noaudio --background --python generate.py -- --seed 0 --task coarse -g desert.gin simple.gin --output_folder outputs/helloworld/coarse + +# Populate unique assets +$BLENDER -noaudio --background --python generate.py -- --seed 0 --task populate fine_terrain -g desert.gin simple.gin --input_folder outputs/helloworld/coarse --output_folder outputs/helloworld/fine + +# Render RGB images +$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames + +# Render again for accurate ground-truth +$BLENDER -noaudio --background --python generate.py -- --seed 0 --task render -g desert.gin simple.gin --input_folder outputs/helloworld/fine --output_folder outputs/helloworld/frames -p render.render_image_func=@flat/render_image +``` + +Output logs should indicate what the code is working on. Use `--debug` for even more detail. After each command completes you can inspect it's `--output_folder` for results, including running `$BLENDER outputs/helloworld/coarse/scene.blend` or similar to view blender files. We hide many meshes by default for viewport stability; to view them, click "Render" or use the UI to unhide them. + +## Generate image(s) in one command + +We provide `tools/manage_datagen_jobs.py`, a utility which runs similar steps automatically. + +``` +python -m tools.manage_datagen_jobs --output_folder outputs/hello_world --num_scenes 1 --specific_seed 0 +--configs desert.gin simple.gin --pipeline_configs local_16GB.gin monocular.gin blender_gt.gin --pipeline_overrides LocalScheduleHandler.use_gpu=False +``` + +This command will repeatedly print summaries of the status of each stage of the pipeline. Please look in `outputs/hello_world/1/logs` for full output logs of the underlying tasks. + +We encourage you to visit [Configuring Infinigen](docs/ConfiguringInfinigen.md) for a breakdown of this command and more advanced usage instructions / example commands. + +See [Extended ground-truth](docs/GroundTruthAnnotations.md) for a guide on using our custom ground-truth extraction system. \ No newline at end of file diff --git a/docs/Installation.md b/docs/Installation.md new file mode 100644 index 000000000..eb4330bba --- /dev/null +++ b/docs/Installation.md @@ -0,0 +1,68 @@ + +# Installation + +Installation is tested and working on the following platforms: + +- Ubuntu 22.04.2 LTS + - GPUs options tested: CPU only, GTX-1080, RTX-2080, RTX-3090, RTX-4090 (Driver 525, CUDA 12.0) + - RAM: 16GB +- MacOS Monterey & Ventura, Apple M1 Pro, 16GB RAM + +We are working on support for rendering with AMD GPUs. Windows users should use [WSL2](https://ubuntu.com/tutorials/install-ubuntu-on-wsl2-on-windows-11-with-gui-support#1-overview). More instructions coming soon. + +
+:warning: Errors with git pull / merge conflicts when migrating from v1.0.0 to v1.0.1 +To properly display open-source line by line git credits for our team, we have switched to a new version of the repo which does not share commit history with the the version available from 6/17/2023 to 6/29/2023 date. We hope this will help open source contributors identify the current "code owner" or person best equipped to support you with issues you encounter with any particular lines of the codebase. + +You will not be able to pull or merge infinigen v1.0.1 into a v1.0.0 repo without significant git expertise. If you have no ongoing changes, we recommend you clone a new copy of the repo. We apologize for any inconvenience, please make an issue if you have problems updating or need help migrating ongoing changes. We understand this change is disruptive, but it is one-time-only and will not occur in future versions. Now it is complete, we intend to iterate rapidly in the coming weeks, please see our [roadmap](https://infinigen.org/roadmap) and [twitter](https://twitter.com/PrincetonVL) for updates. +
+ +**Run these commands to get started** +``` +git clone --recursive https://github.com/princeton-vl/infinigen.git +cd infinigen +conda create --name infinigen python=3.10 +conda activate infinigen +bash install.sh +``` +`install.sh` may take significant time to download Blender and compile all source files. + +Ignore non-fatal warnings. See [Getting Help](#getting-help) for guidelines on posting github issues + +Run the following or add it to your `~/.bashrc` (Linux/WSL) or `~/.bash_profile` (Mac) +``` +# on Linux/WSL +export BLENDER="/PATH/TO/infinigen/blender/blender" +# on MAC +export BLENDER="/PATH/TO/infinigen/Blender.app/Contents/MacOS/Blender" +``` + +
+(Optional) Running Infinigen in a Docker Container + +**Docker on Linux** + +In `/infinigen/` +``` +make docker-build +make docker-setup +make docker-run +``` +To enable CUDA compilation, use `make docker-build-cuda` instead of `make docker-build` + +To run without GPU passthrough use `make docker-run-no-gpu` + +To run without OpenGL ground truth use `docker-run-no-opengl` + +To run without either, use `docker-run-no-gpu-opengl` + +Note: `make docker-setup` can be skipped if not using OpenGL. + +Use `exit` to exit the container and `docker exec -it infinigen bash` to re-enter the container as needed. Remember to `conda activate infinigen` before running scenes. + + +**Docker on Windows** + +Install [WSL2](https://infinigen.org/docs/installation/intro#setup-for-windows) and [Docker Desktop](https://www.docker.com/products/docker-desktop/), with "Use the WSL 2 based engine..." enabled in settings. Keep the Docker Desktop application open while running containers. Then follow instructions as above. + +
\ No newline at end of file diff --git a/images/gt_annotations/Depth_0001_00_00.png b/docs/images/gt_annotations/Depth_0001_00_00.png similarity index 100% rename from images/gt_annotations/Depth_0001_00_00.png rename to docs/images/gt_annotations/Depth_0001_00_00.png diff --git a/images/gt_annotations/OcclusionBoundaries_0001_00_00.png b/docs/images/gt_annotations/OcclusionBoundaries_0001_00_00.png similarity index 100% rename from images/gt_annotations/OcclusionBoundaries_0001_00_00.png rename to docs/images/gt_annotations/OcclusionBoundaries_0001_00_00.png diff --git a/images/gt_annotations/SurfaceNormal_0001_00_00.png b/docs/images/gt_annotations/SurfaceNormal_0001_00_00.png similarity index 100% rename from images/gt_annotations/SurfaceNormal_0001_00_00.png rename to docs/images/gt_annotations/SurfaceNormal_0001_00_00.png diff --git a/images/gt_annotations/blender_rock_2d.png b/docs/images/gt_annotations/blender_rock_2d.png similarity index 100% rename from images/gt_annotations/blender_rock_2d.png rename to docs/images/gt_annotations/blender_rock_2d.png diff --git a/images/gt_annotations/blender_rock_3d.png b/docs/images/gt_annotations/blender_rock_3d.png similarity index 100% rename from images/gt_annotations/blender_rock_3d.png rename to docs/images/gt_annotations/blender_rock_3d.png diff --git a/images/gt_annotations/caves.png b/docs/images/gt_annotations/caves.png similarity index 100% rename from images/gt_annotations/caves.png rename to docs/images/gt_annotations/caves.png diff --git a/images/gt_annotations/petal.png b/docs/images/gt_annotations/petal.png similarity index 100% rename from images/gt_annotations/petal.png rename to docs/images/gt_annotations/petal.png diff --git a/images/gt_annotations/warped_rocks.png b/docs/images/gt_annotations/warped_rocks.png similarity index 100% rename from images/gt_annotations/warped_rocks.png rename to docs/images/gt_annotations/warped_rocks.png diff --git a/images/Depth0048_00_00.png b/docs/images/hello_world/Depth0048_00_00.png similarity index 100% rename from images/Depth0048_00_00.png rename to docs/images/hello_world/Depth0048_00_00.png diff --git a/images/Image0048_00_00.png b/docs/images/hello_world/Image0048_00_00.png similarity index 100% rename from images/Image0048_00_00.png rename to docs/images/hello_world/Image0048_00_00.png diff --git a/images/InstanceSegmentation_0001_00_00.png b/docs/images/hello_world/InstanceSegmentation_0001_00_00.png similarity index 100% rename from images/InstanceSegmentation_0001_00_00.png rename to docs/images/hello_world/InstanceSegmentation_0001_00_00.png diff --git a/images/SurfaceNormal_0001_00_00.png b/docs/images/hello_world/SurfaceNormal_0001_00_00.png similarity index 100% rename from images/SurfaceNormal_0001_00_00.png rename to docs/images/hello_world/SurfaceNormal_0001_00_00.png diff --git a/docs/images/individual_assets/chunkyrock.png b/docs/images/individual_assets/chunkyrock.png new file mode 100644 index 000000000..9ad1e9806 Binary files /dev/null and b/docs/images/individual_assets/chunkyrock.png differ diff --git a/docs/images/individual_assets/coral.png b/docs/images/individual_assets/coral.png new file mode 100644 index 000000000..1b693397d Binary files /dev/null and b/docs/images/individual_assets/coral.png differ diff --git a/docs/images/individual_assets/seashells.png b/docs/images/individual_assets/seashells.png new file mode 100644 index 000000000..fa9c44a61 Binary files /dev/null and b/docs/images/individual_assets/seashells.png differ diff --git a/images/video_thumbnail.png b/docs/images/video_thumbnail.png similarity index 100% rename from images/video_thumbnail.png rename to docs/images/video_thumbnail.png diff --git a/worldgen/config/base.gin b/worldgen/config/base.gin index 24e1772c5..8daa82582 100644 --- a/worldgen/config/base.gin +++ b/worldgen/config/base.gin @@ -13,8 +13,8 @@ save_obj_and_instances.output_folder="saved_mesh.obj" util.logging.create_text_file.log_dir = %LOG_DIR -compose_scene.inview_distance = 70 placement.populate_all.dist_cull = 70 +compose_scene.inview_distance = 70 compose_scene.near_distance = 20 compose_scene.center_distance = 35 @@ -157,12 +157,11 @@ camera.camera_pose_proposal.altitude = ("weighted_choice", camera.camera_pose_proposal.pitch = ("clip_gaussian", 90, 30, 20, 160) -camera.spawn_camera_rigs.n_camera_rigs = 1 - # WARNING: Large camera rig translations or rotations require special handling. # if your cameras are not all approximately forward facing within a few centimeters, you must either: # - configure the pipeline to generate assets / terrain for each camera separately, rather than sharing it between the whole rig # - or, treat your camera rig as multiple camera rigs each with one camera, and implement code to positon them correctly +camera.spawn_camera_rigs.n_camera_rigs = 1 camera.spawn_camera_rigs.camera_rig_config = [ {'loc': (0, 0, 0), 'rot_euler': (0, 0, 0)}, {'loc': (0.075, 0, 0), 'rot_euler': (0, 0, 0)} diff --git a/worldgen/config/no_assets.gin b/worldgen/config/disable_assets/no_assets.gin similarity index 100% rename from worldgen/config/no_assets.gin rename to worldgen/config/disable_assets/no_assets.gin diff --git a/worldgen/config/no_creatures.gin b/worldgen/config/disable_assets/no_creatures.gin similarity index 100% rename from worldgen/config/no_creatures.gin rename to worldgen/config/disable_assets/no_creatures.gin diff --git a/worldgen/config/no_particles.gin b/worldgen/config/disable_assets/no_particles.gin similarity index 100% rename from worldgen/config/no_particles.gin rename to worldgen/config/disable_assets/no_particles.gin diff --git a/worldgen/config/no_rocks.gin b/worldgen/config/disable_assets/no_rocks.gin similarity index 100% rename from worldgen/config/no_rocks.gin rename to worldgen/config/disable_assets/no_rocks.gin diff --git a/worldgen/config/dev.gin b/worldgen/config/performance/dev.gin similarity index 100% rename from worldgen/config/dev.gin rename to worldgen/config/performance/dev.gin diff --git a/worldgen/config/fast_terrain_assets.gin b/worldgen/config/performance/fast_terrain_assets.gin similarity index 100% rename from worldgen/config/fast_terrain_assets.gin rename to worldgen/config/performance/fast_terrain_assets.gin diff --git a/worldgen/config/high_quality_terrain.gin b/worldgen/config/performance/high_quality_terrain.gin similarity index 100% rename from worldgen/config/high_quality_terrain.gin rename to worldgen/config/performance/high_quality_terrain.gin diff --git a/worldgen/config/reuse_terrain_assets.gin b/worldgen/config/performance/reuse_terrain_assets.gin similarity index 100% rename from worldgen/config/reuse_terrain_assets.gin rename to worldgen/config/performance/reuse_terrain_assets.gin diff --git a/worldgen/config/performance/simple.gin b/worldgen/config/performance/simple.gin new file mode 100644 index 000000000..5320ec9bc --- /dev/null +++ b/worldgen/config/performance/simple.gin @@ -0,0 +1,5 @@ +include 'config/performance/dev.gin' +include 'config/disable_assets/no_creatures.gin' +include 'config/performance/fast_terrain_assets.gin' +run_erosion.n_iters = [1,1] +full/render_image.num_samples = 64 \ No newline at end of file diff --git a/worldgen/config/scene_types/arctic.gin b/worldgen/config/scene_types/arctic.gin index 801d37624..d6ca4c7bc 100644 --- a/worldgen/config/scene_types/arctic.gin +++ b/worldgen/config/scene_types/arctic.gin @@ -29,7 +29,7 @@ scene.warped_rocks_chance = 0 scene.ground_ice_chance = 1 scene.waterbody_chance = 1 -include 'config/no_assets.gin' +include 'config/disable_assets/no_assets.gin' compose_scene.wind_chance = 0.5 compose_scene.turbulence_chance = 0.5 diff --git a/worldgen/config/scene_types/snowy_mountain.gin b/worldgen/config/scene_types/snowy_mountain.gin index 6d9d2192b..8c8884150 100644 --- a/worldgen/config/scene_types/snowy_mountain.gin +++ b/worldgen/config/scene_types/snowy_mountain.gin @@ -1,4 +1,4 @@ -include 'config/no_assets.gin' +include 'config/disable_assets/no_assets.gin' surface.registry.rock_collection = [ ('mountain', 1), @@ -6,7 +6,7 @@ surface.registry.rock_collection = [ surface.registry.mountain_collection = [ ('mountain', 1), ] - + compose_scene.snow_particles_chance = 0.5 shader_atmosphere.density = 0 diff --git a/worldgen/config/simple.gin b/worldgen/config/simple.gin deleted file mode 100644 index 099cf47a0..000000000 --- a/worldgen/config/simple.gin +++ /dev/null @@ -1,5 +0,0 @@ -include 'config/dev.gin' -include 'config/no_creatures.gin' -include 'config/fast_terrain_assets.gin' -run_erosion.n_iters = [1,1] -full/render_image.num_samples = 64 \ No newline at end of file diff --git a/worldgen/core.py b/worldgen/core.py index 2aaefc5e2..98c457067 100644 --- a/worldgen/core.py +++ b/worldgen/core.py @@ -96,10 +96,10 @@ def populate_scene( populated = {} populated['trees'] = p.run_stage('populate_trees', use_chance=False, default=[], - fn=lambda: placement.populate_all(TreeFactory, camera, season=season, vis_cull=4, dist_cull=70))#, + fn=lambda: placement.populate_all(TreeFactory, camera, season=season, vis_cull=4))#, #meshing_camera=camera, adapt_mesh_method='subdivide', cam_meshing_max_dist=8)) populated['boulders'] = p.run_stage('populate_boulders', use_chance=False, default=[], - fn=lambda: placement.populate_all(boulder.BoulderFactory, camera, vis_cull=3, dist_cull=70))#, + fn=lambda: placement.populate_all(boulder.BoulderFactory, camera, vis_cull=3))#, #meshing_camera=camera, adapt_mesh_method='subdivide', cam_meshing_max_dist=8)) p.run_stage('populate_bushes', use_chance=False, fn=lambda: placement.populate_all(BushFactory, camera, vis_cull=1, adapt_mesh_method='subdivide')) @@ -396,19 +396,15 @@ def apply_gin_configs(args, scene_seed, skip_unknown=False): args.gin_config = [scene_type] + args.gin_config def find_config(g): - as_figure_type = f'config/figure_types/{g}.gin' - if os.path.exists(as_figure_type): - return as_figure_type - as_scene_type = f'config/scene_types/{g}.gin' - if os.path.exists(as_scene_type): - return as_scene_type - as_base = f'config/{g}.gin' - if os.path.exists(as_base): - return as_base - raise ValueError(f'Couldn not locate {g} in either config/ or config/scene_types') + for p in Path('config').glob('**/*.gin'): + if p.parts[-1] == g: + return p + if p.parts[-1] == f'{g}.gin': + return p + raise ValueError(f'Couldn not locate {g} or {g}.gin in anywhere config/**') bindings = sanitize_gin_override(args.gin_param) - confs = [find_config(g) for g in ['base'] + args.gin_config] + confs = [find_config(g) for g in ['base.gin'] + args.gin_config] gin.parse_config_files_and_bindings(confs, bindings=bindings, skip_unknown=skip_unknown) def main( diff --git a/worldgen/tools/manage_datagen_jobs.py b/worldgen/tools/manage_datagen_jobs.py index dbcf3b64e..3df49ae6f 100644 --- a/worldgen/tools/manage_datagen_jobs.py +++ b/worldgen/tools/manage_datagen_jobs.py @@ -555,8 +555,16 @@ def init_db(args, inorder_seeds=False, enumerate_scenetypes=[None]): def update_symlink(scene_folder, scenes): for new_name, scene in scenes: - std_out = scene_folder / "logs" / f"{scene.job_id}_0_log.out" + + if scene == 'MARK_AS_SUCCEEDED': + continue + elif isinstance(scene, str): + raise ValueError(f'Failed due to {scene=}') + to = scene_folder / "logs" / f"{new_name}.out" + + std_out = scene_folder / "logs" / f"{scene.job_id}_0_log.out" + if os.path.islink(to): os.unlink(to) os.unlink(scene_folder / "logs" / f"{new_name}.err") @@ -982,24 +990,98 @@ def test_upload(args): slurm_available = (which("sbatch") is not None) parser = argparse.ArgumentParser() # to guarantee that the render scenes finish, try render_image.time_limit=2000 - parser.add_argument('--blender_path', type=str, default=None) - parser.add_argument('-o', '--output_folder', type=Path, required=True) - parser.add_argument('--num_scenes', type=int, default=1) - parser.add_argument('--meta_seed', type=int, default=None) - parser.add_argument('--specific_seed', default=None, nargs='+', help="It will cast to an int if it's a number, otherwise str") - parser.add_argument('--use_existing', action='store_true') - - parser.add_argument('--warmup_sec', type=float, default=0) - parser.add_argument('--cleanup', type=str, choices=['all', 'big_files', 'none', 'except_crashed'], default='none') - parser.add_argument('--remove_write', action='store_true') - parser.add_argument('--upload', action='store_true') - - parser.add_argument('--configs', nargs='*', default=[]) - parser.add_argument('-p', '--override', nargs='+', type=str, default=[], help="gin-config settings to override. FYI all scenes will use the same overrides.") - - parser.add_argument('--wandb_mode', type=str, default='disabled', choices=['online', 'offline', 'disabled']) - parser.add_argument('--pipeline_configs', type=str, nargs='+', default=["allcs" if slurm_available else "local"]) - parser.add_argument('--pipeline_overrides', nargs='+', type=str, default=[], help="pipeline gin-config settings to override") + parser.add_argument( + '-o', + '--output_folder', + type=Path, + required=True + ) + parser.add_argument( + '--blender_path', + type=str, + default=None, + help="Full path to a `blender` executable with all dependencies installed. If set to None, the system will use the $BLENDER environment variable" + ) + parser.add_argument( + '--num_scenes', + type=int, + default=1, + help="Number of scenes to attempt before terminating" + ) + parser.add_argument( + '--meta_seed', + type=int, + default=None, + help="What seed should be used to determine the random seeds of each scene? Leave as None unless deliberately replicat" + ) + parser.add_argument( + '--specific_seed', + default=None, + nargs='+', + help="The default, None, will choose a random seed per scene. Otherwise, all scenes will have the specified seed. Interpreted as an integer if possible." + ) + parser.add_argument( + '--use_existing', + action='store_true', + help="If set, then assume output_folder is an existing folder from a terminated run, and make a best-possible-effort to resume from where it left off" + ) + parser.add_argument( + '--warmup_sec', + type=float, + default=0, + help="Perform a staggered start over the specified period, so that jobs dont sync up or all write to disk at similar times." + ) + parser.add_argument( + '--cleanup', + type=str, + choices=['all', 'big_files', 'none', 'except_crashed'], + default='none', + help="What files should be cleaned up by the manager as it runs?" + ) + parser.add_argument( + '--remove_write', + action='store_true', + help="If set, remove Unix write permissions from the generated files, to prevent a user accidentally deleting data." + ) + parser.add_argument( + '--upload', + action='store_true', + help="If set, attempt to upload scenes as tarballs to GDrive of SMB depending on configs. See `upload_util.py` for more details before using." + ) + parser.add_argument( + '--configs', + nargs='*', + default=[], + help="List of gin config names to pass through to all underlying scene generation jobs." + ) + parser.add_argument( + '-p', + '--override', + nargs='+', + type=str, + default=[], + help="List of gin overrides to pass through to all underlying scene generation jobs" + ) + parser.add_argument( + '--wandb_mode', + type=str, + default='disabled', + choices=['online', 'offline', 'disabled'], + help="Mode kwarg for wandb.init(). Set up wandb before use." + ) + parser.add_argument( + '--pipeline_configs', + type=str, + nargs='+', + help="List of gin config names from tools/pipeline_configs to configure this execution" + ) + parser.add_argument( + '--pipeline_overrides', + nargs='+', + type=str, + default=[], + help="List of gin overrides to configure this execution", + ) parser.add_argument('-d', '--debug', action="store_const", dest="loglevel", const=logging.DEBUG, default=logging.INFO) parser.add_argument( '-v', '--verbose', action="store_const", dest="loglevel", const=logging.INFO) args = parser.parse_args() @@ -1007,7 +1089,7 @@ def test_upload(args): envvar = 'INFINIGEN_ASSET_FOLDER' if not args.upload and args.cleanup == 'all': - raise ValueError(f'Pipeline is configured {args.upload=} {args.cleanup=} --- no output would be preserved') + raise ValueError(f'Pipeline is configured with {args.cleanup=} yet {args.upload=} --- no output would be preserved') global BLENDER_PATH if args.blender_path is None: @@ -1031,8 +1113,14 @@ def test_upload(args): random.seed(args.meta_seed) np.random.seed(args.meta_seed) - find_gin = lambda n: os.path.join("tools", "pipeline_configs", f"{n}.gin") - configs = [find_gin(n) for n in args.pipeline_configs] + def find_config(g): + for p in Path('tools/pipeline_configs').glob('**/*.gin'): + if p.parts[-1] == g: + return p + if p.parts[-1] == f'{g}.gin': + return p + raise ValueError(f'Couldn not locate {g} or {g}.gin in anywhere pipeline_configs/**') + configs = [find_config(n) for n in args.pipeline_configs] for c in configs: assert os.path.exists(c), c bindings = args.pipeline_overrides diff --git a/worldgen/tools/pipeline_configs/compute_platform/local_128GB.gin b/worldgen/tools/pipeline_configs/compute_platform/local_128GB.gin new file mode 100644 index 000000000..d73798fa0 --- /dev/null +++ b/worldgen/tools/pipeline_configs/compute_platform/local_128GB.gin @@ -0,0 +1,5 @@ +include 'tools/pipeline_configs/compute_platform/local_256GB.gin' + +manage_datagen_jobs.num_concurrent=8 + + diff --git a/worldgen/tools/pipeline_configs/compute_platform/local_16GB.gin b/worldgen/tools/pipeline_configs/compute_platform/local_16GB.gin new file mode 100644 index 000000000..3ee5e8d96 --- /dev/null +++ b/worldgen/tools/pipeline_configs/compute_platform/local_16GB.gin @@ -0,0 +1,6 @@ +include 'tools/pipeline_configs/compute_platform/local_256GB.gin' + +manage_datagen_jobs.num_concurrent=1 + + + diff --git a/worldgen/tools/pipeline_configs/local_256GB.gin b/worldgen/tools/pipeline_configs/compute_platform/local_256GB.gin similarity index 100% rename from worldgen/tools/pipeline_configs/local_256GB.gin rename to worldgen/tools/pipeline_configs/compute_platform/local_256GB.gin diff --git a/worldgen/tools/pipeline_configs/compute_platform/local_64GB.gin b/worldgen/tools/pipeline_configs/compute_platform/local_64GB.gin new file mode 100644 index 000000000..9abdc054b --- /dev/null +++ b/worldgen/tools/pipeline_configs/compute_platform/local_64GB.gin @@ -0,0 +1,5 @@ +include 'tools/pipeline_configs/compute_platform/local_256GB.gin' + +manage_datagen_jobs.num_concurrent=3 + + diff --git a/worldgen/tools/pipeline_configs/slurm.gin b/worldgen/tools/pipeline_configs/compute_platform/slurm.gin similarity index 96% rename from worldgen/tools/pipeline_configs/slurm.gin rename to worldgen/tools/pipeline_configs/compute_platform/slurm.gin index ba961ae53..cd9686cb5 100644 --- a/worldgen/tools/pipeline_configs/slurm.gin +++ b/worldgen/tools/pipeline_configs/compute_platform/slurm.gin @@ -18,6 +18,7 @@ queue_coarse.cpus = 4 queue_coarse.hours = 48 queue_coarse.slurm_account = %PARTITION queue_coarse.submit_cmd = @slurm_submit_cmd +queue_coarse.exclude_gpus = ['a6000', 'rtx_3090'] # Fine terrain queue_fine_terrain.mem_gb = 24 @@ -33,6 +34,7 @@ queue_populate.hours = 24 queue_populate.slurm_account = %PARTITION queue_populate.submit_cmd = @slurm_submit_cmd renderbackup/queue_populate.mem_gb = 24 +queue_populate.exclude_gpus = ['a6000', 'rtx_3090'] # Rendering queue_render.submit_cmd = @slurm_submit_cmd diff --git a/worldgen/tools/pipeline_configs/monocular.gin b/worldgen/tools/pipeline_configs/data_schema/monocular.gin similarity index 100% rename from worldgen/tools/pipeline_configs/monocular.gin rename to worldgen/tools/pipeline_configs/data_schema/monocular.gin diff --git a/worldgen/tools/pipeline_configs/data_schema/monocular_flow.gin b/worldgen/tools/pipeline_configs/data_schema/monocular_flow.gin new file mode 100644 index 000000000..b763ca944 --- /dev/null +++ b/worldgen/tools/pipeline_configs/data_schema/monocular_flow.gin @@ -0,0 +1,4 @@ +include 'tools/pipeline_configs/data_schema/monocular.gin' +iterate_scene_tasks.frame_range=(1, 2) + + diff --git a/worldgen/tools/pipeline_configs/monocular_video.gin b/worldgen/tools/pipeline_configs/data_schema/monocular_video.gin similarity index 100% rename from worldgen/tools/pipeline_configs/monocular_video.gin rename to worldgen/tools/pipeline_configs/data_schema/monocular_video.gin diff --git a/worldgen/tools/pipeline_configs/stereo.gin b/worldgen/tools/pipeline_configs/data_schema/stereo.gin similarity index 100% rename from worldgen/tools/pipeline_configs/stereo.gin rename to worldgen/tools/pipeline_configs/data_schema/stereo.gin diff --git a/worldgen/tools/pipeline_configs/stereo_1h_jobs.gin b/worldgen/tools/pipeline_configs/data_schema/stereo_1h_jobs.gin similarity index 100% rename from worldgen/tools/pipeline_configs/stereo_1h_jobs.gin rename to worldgen/tools/pipeline_configs/data_schema/stereo_1h_jobs.gin diff --git a/worldgen/tools/pipeline_configs/data_schema/stereo_video.gin b/worldgen/tools/pipeline_configs/data_schema/stereo_video.gin new file mode 100644 index 000000000..8c92326df --- /dev/null +++ b/worldgen/tools/pipeline_configs/data_schema/stereo_video.gin @@ -0,0 +1,2 @@ +include 'tools/pipeline_configs/data_schema/monocular_video.gin' +iterate_scene_tasks.cam_id_ranges = [1, 2] diff --git a/worldgen/tools/pipeline_configs/local_128GB.gin b/worldgen/tools/pipeline_configs/local_128GB.gin deleted file mode 100644 index cb5128800..000000000 --- a/worldgen/tools/pipeline_configs/local_128GB.gin +++ /dev/null @@ -1,5 +0,0 @@ -include 'tools/pipeline_configs/local_256GB.gin' - -manage_datagen_jobs.num_concurrent=8 - - diff --git a/worldgen/tools/pipeline_configs/local_16GB.gin b/worldgen/tools/pipeline_configs/local_16GB.gin deleted file mode 100644 index d46c51157..000000000 --- a/worldgen/tools/pipeline_configs/local_16GB.gin +++ /dev/null @@ -1,6 +0,0 @@ -include 'tools/pipeline_configs/local_256GB.gin' - -manage_datagen_jobs.num_concurrent=1 - - - diff --git a/worldgen/tools/pipeline_configs/local_64GB.gin b/worldgen/tools/pipeline_configs/local_64GB.gin deleted file mode 100644 index 76088aac6..000000000 --- a/worldgen/tools/pipeline_configs/local_64GB.gin +++ /dev/null @@ -1,5 +0,0 @@ -include 'tools/pipeline_configs/local_256GB.gin' - -manage_datagen_jobs.num_concurrent=3 - - diff --git a/worldgen/tools/pipeline_configs/monocular_flow.gin b/worldgen/tools/pipeline_configs/monocular_flow.gin deleted file mode 100644 index cc5e6e13c..000000000 --- a/worldgen/tools/pipeline_configs/monocular_flow.gin +++ /dev/null @@ -1,4 +0,0 @@ -include 'tools/pipeline_configs/monocular.gin' -iterate_scene_tasks.frame_range=(1, 2) - - diff --git a/worldgen/tools/pipeline_configs/stereo_video.gin b/worldgen/tools/pipeline_configs/stereo_video.gin deleted file mode 100644 index 46cea7e19..000000000 --- a/worldgen/tools/pipeline_configs/stereo_video.gin +++ /dev/null @@ -1,2 +0,0 @@ -include 'tools/pipeline_configs/monocular_video.gin' -iterate_scene_tasks.cam_id_ranges = [1, 2] diff --git a/worldgen/tools/render_video_final.sh b/worldgen/tools/render_video_final.sh new file mode 100644 index 000000000..bbc7020b7 --- /dev/null +++ b/worldgen/tools/render_video_final.sh @@ -0,0 +1,4 @@ +HOSTFIRST=$(hostname | tr "." "\n" | head -n 1) +JOBNAME=$(date '+%m_%d_%H_%M').$HOSTFIRST.$1 +python -m tools.manage_datagen_jobs --blender_path ../blender/blender --output_folder outputs/$JOBNAME --num_scenes 500 \ + --pipeline_config slurm stereo_video cuda_terrain opengl_gt --wandb_mode online --cleanup big_files --warmup_sec 60000 --config trailer high_quality_terrain --upload