diff --git a/README.md b/README.md index 0e553a487..f1b3abde1 100644 --- a/README.md +++ b/README.md @@ -113,6 +113,7 @@ Conference on Computer Vision and Pattern Recognition (CVPR) 2024 - [Downloading pre-generated data](docs/PreGeneratedData.md) - [Generating individual assets](docs/GeneratingIndividualAssets.md) - [Exporting to external fileformats (OBJ, OpenUSD, etc)](docs/ExportingToExternalFileFormats.md) +- [Add external assets to indoor scenes](docs/StaticAssets.md) - [Extended ground-truth](docs/GroundTruthAnnotations.md) - [Implementing new materials & assets](docs/ImplementingAssets.md) - [Generating fluid simulations](docs/GeneratingFluidSimulations.md) diff --git a/docs/CHANGELOG.md b/docs/CHANGELOG.md index 779fcd54f..47d3a3fdb 100644 --- a/docs/CHANGELOG.md +++ b/docs/CHANGELOG.md @@ -91,3 +91,6 @@ v1.7.0 v1.7.1 - Bugfix fine terrain in arctic scenes + +v.1.8.0 +- Implement tools for importing external assets into Indoors diff --git a/docs/StaticAssets.md b/docs/StaticAssets.md new file mode 100644 index 000000000..0b71acc60 --- /dev/null +++ b/docs/StaticAssets.md @@ -0,0 +1,197 @@ +# Static Assets: Import External Assets to Infinigen Indoors + +In this guide, we will show you how to import external (static) assets into Infinigen. This is useful if you want to create scenes with categories of assets that are not covered by Infinigen (say sculptures, etc), or if you just want to add more variety to your scenes with custom assets. + +:warning: This guide is not the only way to add assets to an Infinigen scene. Infinigen works with Blender scenes, so you can always use Blender's [import ops](https://docs.blender.org/api/current/bpy.ops.import_scene.html) to add objects. The instructions below go beyond this by showing how to integrate with the placement system, IE how to make the assets appear at sensible random locations. + +## Download the Objects + +We will get started by downloading the objects we want to import. For this example, we will use a shelf, a leather sofa, and a table from [Objaverse](https://objaverse.allenai.org/). You can use any objects you like, as long as it is one of the following formats: +- .dae +- .abc +- .usd +- .obj +- .ply +- .stl +- .fbx +- .glb +- .gltf +- .blend + +I would recommend using .glb or .gltf files since they are the most extensively tested. + +1. Download the following assets in .glb format from Sketchfab: + + - [Iron Shelf](https://sketchfab.com/3d-models/iron-shelf-fd0cd420ffe04ac08174926f6b175d3f) (Attributed to Thunder and is licensed under CC BY-SA 4.0. We use this asset with modification, i.e. merge parts, scale). + + - [Office Couch](https://sketchfab.com/3d-models/office-couch-ca63db1db205476fa6f54e1603b7d15d) (Attributed to Virenimation and is licensed under CC BY 4.0. We use this asset with modification, i.e. merge parts, scale). + + - [Table](https://sketchfab.com/3d-models/de-table-63e4d8faac73435fa7e9e929baa2c175) (Attributed to DeCloud and is licensed under CC BY 4.0. We use this asset with modification, i.e. merge parts, scale). + +

+ + + +

+ +2. Create a folder for each category you want to import in `infinigen/assets/static_assets/source`. That is, create the folders `infinigen/assets/static_assets/Shelf`, `infinigen/assets/static_assets/Sofa`, and `infinigen/assets/static_assets/Table`. + +```bash +mkdir infinigen/assets/static_assets/source +mkdir infinigen/assets/static_assets/source/Shelf +mkdir infinigen/assets/static_assets/source/Sofa +mkdir infinigen/assets/static_assets/source/Table +``` + +3. Place the downloaded .glb files in the corresponding folders. For this example, you should have three folders with one .glb file each. + +Now, whenever we tell Infinigen to import a static Shelf asset in the scene, it will look for the .glb file in the Shelf folder we created. If you have multiple objects within the same category, Infinigen will randomly choose one of them each time that it wants to place in the scene. You can also put objects of different formats in the same folder: `shelf1.glb, shelf2.obj, shelf3.fbx`, etc. + +NOTE: Objaverse supports downloads using python [API](https://colab.research.google.com/drive/15XpZMjrHXuky0IgBbXcsUtb_0g-XWYmN?usp=sharing) if you want to download a large number of objects. + + +## Create Static Asset Category + +Now, we just need to define `Static{CategoryName}Factory = static_category_factory({CategoryName})`. Add the following lines to `infinigen/assets/static_assets/static_category.py` (This step is already done for this tutorial): + +![alt text](images/static_assets/image.jpg) + +Add `Static{CategoryName}Factory` to `infinigen/assets/static_assets/__init__.py` so that we can import it later. Again, this step is already done for our example: + +![alt text](images/static_assets/image2.jpg) + +IMPORTANT: You must make sure that +1. The objects are a reasonable size. This is because there is no way for Infinigen to infer what the sizes of the objects should be, so it will use the sizes of external objects by default. If the objects are too big or too small, they will look out of place in the scene or Infinigen will fail to place them. To prevent this, you should specify what you want the dimensions to be in meters as we did with `z_dim = 2` above. You should specify only one dimension, and Infinigen will scale the object to satisfy that. + +2. The front of the object is facing +x, the back is facing -x, the bottom is facing -z, and the top is facing +z. You can do this by passing `rotation_euler = (x,y,z)` in Euler angles to `static_category_factory`. This information is important, for instance, when placing a sofa against the wall. + +3. Make sure that `CategoryName` matches the name of the folder you created in the `static_assets/source` folder. + +Here's an example of bad dimensions and orientation (left image); and good dimensions and orientation (right image): + +

+ + +

+ + +If you want to add more categories, just add more lines with `{CategoryName}` as the name of the category you want to import. + +## Define Semantics + +Infinigen allows the user to specify high-level semantics for the objects in the scene. These semantics are then used to define high-level constraints. For example, we want to say that our static shelf factory is a type of storage unit, which will be placed against the wall, and there will be a bunch of objects on top of it. In general, if you want your static object factory to be treated like an existing asset factory, you can just imitate the semantics of the existing asset factory. Let's demonstrate this idea by defining semantics for our static shelf. We go to `infinigen_examples/indoor_asset_semantics.py` and search for `LargeShelfFactory`. We see that it is used as `Semantics.Storage` and `Semantics.AssetPlaceholderForChildren`. We want our static shelf to be used as a storage unit as well, so we add a line for our new static factory: +![alt text](images/static_assets/image3.jpg) + +Similarly, we add `StaticShelfFactory` to `Semantics.AssetPlaceholderForChildren`. This will replace the placeholder bounding box for the shelf before placing the small objects. +![alt text](images/static_assets/image7.jpg) + +The semantics for the sofa and the table are analogous. We just define the same semantics as the existing asset factories that are similar to our static ones: + +![alt text](images/static_assets/image4.jpg) + +![alt text](images/static_assets/image5.jpg) + +![alt text](images/static_assets/image6.jpg) + +If your category is not similar to any existing category, you would need to think about it a little bit more and define your own semantics. For example, I found that the sofa semantics work well for vending machine as they are both placed against the wall, etc. + +## Add Constraints + +The last step is to add constraints for our static assets. We want to make sure that the shelf is placed against the wall, the sofa is placed in the living room, and the table is placed in the dining room, etc. Luckily our new static assets are similar to existing assets, so we can just replace the existing constraints with our new static assets. We go to `infinigen_examples/indoor_constraint_examples.py` and search for `LargeShelfFactory`, `SofaFactory`, and `TableDiningFactory`. We just replace these constraints with our new static assets: + +![alt text](images/static_assets/image8.jpg) + +![alt text](images/static_assets/image9.jpg) + +![alt text](images/static_assets/image10.jpg) + +If you have some new asset that does not behave like any of the existing assets, you would need to define new constraints. + + +## Generate Scene + +And, that's it! You can now generate a scene with your new static assets. We just use the regular Infinigen commands: + +```bash +python -m infinigen_examples.generate_indoors --seed 0 --task coarse --output_folder outputs/indoors/coarse0 -g fast_solve.gin singleroom.gin -p compose_indoors.terrain_enabled=False restrict_solving.restrict_parent_rooms=\[\"LivingRoom\"\] +``` + +![](images/static_assets/untitled8.jpg) + +```bash +python -m infinigen_examples.generate_indoors --seed 1 --task coarse --output_folder outputs/indoors/coarse1 -g fast_solve.gin singleroom.gin -p compose_indoors.terrain_enabled=False restrict_solving.restrict_parent_rooms=\[\"LivingRoom\"\] +``` +![](images/static_assets/untitled9.jpg) + +```bash +python -m infinigen_examples.generate_indoors --seed 11 --task coarse --output_folder outputs/indoors/coarse0dining -g fast_solve.gin singleroom.gin -p compose_indoors.terrain_enabled=False restrict_solving.restrict_parent_rooms=\[\"DiningRoom\"\] +``` +![](images/static_assets/untitled11.jpg) + +You can see that our new static assets are placed in the scene. What's more is that they interact with the existing Infinigen assets. For example, we have a bowl of fruit and a glass on top of the table. + +## Modify Objects with Code (Optional) + +If you want to post-process the imported static objects or customize the object creation, you can define your own static category class similar to `StaticCategoryFactory` in `infinigen/assets/static_assets/static_category.py`. For instance, you can add a `finalize_assets(self, assets)` function to your class to post-process the imported assets. + +## Support Surfaces (Optional) + +A shelf should have small objects placed on it. But, how can Infinigen know where to place these objects? The default is that the top surface of the object is used as the surface for placing objects. For instance, in the above example, we saw that some objects were automatically placed on the table even though we did not specify where to place them. A shelf has multiple surfaces, so we need to tag these surfaces as so-called "support surfaces". This is very easy to do in the case where the support surfaces are the distinct planes along the z direction, as is the case with shelves. We just enable the option `StaticShelfFactory = static_category_factory("Shelf", tag_support=True)` in `infinigen/assets/static_assets/static_category.py`: + +![alt text](images/static_assets/image.jpg) + +Now when we generate the scene, we see that the objects are placed on all surfaces of the shelf: + +![alt text](images/static_assets/shelf_support.jpg) + +## Summary for arbitrary objects + +Let us summarize all the steps you would need to follow to import arbitrary objects into Infinigen: + +1. Download the objects in one of the supported formats. + +2. Create a folder for this object category in `infinigen/assets/static_assets/source`. E.g. + +```bash +mkdir infinigen/assets/static_assets/source/MyCategory +``` + +3. Place the downloaded objects in the folder you created. + +4. Add a line in `infinigen/assets/static_assets/static_category.py` to define the factory for this category. E.g. + +```python +StaticMyCategoryFactory = static_category_factory("MyCategory") +``` + +5. Add a line in `infinigen/assets/static_assets/__init__.py` to import the factory from other files. + +6. Define the semantics for the objects in `infinigen_examples/indoor_asset_semantics.py`. E.g. + +```python +used_as[Semantics.Furniture] = {... + static_assets.StaticMyCategoryFactory} +``` + +7. Define the constraints for the objects in `infinigen_examples/indoor_constraint_examples.py`. E.g. + +```python +my_cat_against_wall = wallfurn[static_assets.StaticMyCategoryFactory] +... +``` + +8. Generate the scene using the regular Infinigen commands. + +## Some Other Examples + +Now that you know how to import static assets, you can import all kinds of different objects. Here are some creative examples: + +

+ + + + + + +

+ diff --git a/docs/images/static_assets/couch.jpg b/docs/images/static_assets/couch.jpg new file mode 100644 index 000000000..952142434 Binary files /dev/null and b/docs/images/static_assets/couch.jpg differ diff --git a/docs/images/static_assets/couch_edit.jpg b/docs/images/static_assets/couch_edit.jpg new file mode 100644 index 000000000..fc7df691d Binary files /dev/null and b/docs/images/static_assets/couch_edit.jpg differ diff --git a/docs/images/static_assets/couch_x.jpg b/docs/images/static_assets/couch_x.jpg new file mode 100644 index 000000000..11cc7bb1a Binary files /dev/null and b/docs/images/static_assets/couch_x.jpg differ diff --git a/docs/images/static_assets/image.jpg b/docs/images/static_assets/image.jpg new file mode 100644 index 000000000..008d4af62 Binary files /dev/null and b/docs/images/static_assets/image.jpg differ diff --git a/docs/images/static_assets/image10.jpg b/docs/images/static_assets/image10.jpg new file mode 100644 index 000000000..5a65041b9 Binary files /dev/null and b/docs/images/static_assets/image10.jpg differ diff --git a/docs/images/static_assets/image11.jpg b/docs/images/static_assets/image11.jpg new file mode 100644 index 000000000..c713676f3 Binary files /dev/null and b/docs/images/static_assets/image11.jpg differ diff --git a/docs/images/static_assets/image2.jpg b/docs/images/static_assets/image2.jpg new file mode 100644 index 000000000..e36d34b34 Binary files /dev/null and b/docs/images/static_assets/image2.jpg differ diff --git a/docs/images/static_assets/image3.jpg b/docs/images/static_assets/image3.jpg new file mode 100644 index 000000000..87b6b1f70 Binary files /dev/null and b/docs/images/static_assets/image3.jpg differ diff --git a/docs/images/static_assets/image4.jpg b/docs/images/static_assets/image4.jpg new file mode 100644 index 000000000..f85141594 Binary files /dev/null and b/docs/images/static_assets/image4.jpg differ diff --git a/docs/images/static_assets/image5.jpg b/docs/images/static_assets/image5.jpg new file mode 100644 index 000000000..e7a1f8945 Binary files /dev/null and b/docs/images/static_assets/image5.jpg differ diff --git a/docs/images/static_assets/image6.jpg b/docs/images/static_assets/image6.jpg new file mode 100644 index 000000000..314fe0eae Binary files /dev/null and b/docs/images/static_assets/image6.jpg differ diff --git a/docs/images/static_assets/image7.jpg b/docs/images/static_assets/image7.jpg new file mode 100644 index 000000000..11363dbed Binary files /dev/null and b/docs/images/static_assets/image7.jpg differ diff --git a/docs/images/static_assets/image8.jpg b/docs/images/static_assets/image8.jpg new file mode 100644 index 000000000..4e5ea1e82 Binary files /dev/null and b/docs/images/static_assets/image8.jpg differ diff --git a/docs/images/static_assets/image9.jpg b/docs/images/static_assets/image9.jpg new file mode 100644 index 000000000..5af630ec9 Binary files /dev/null and b/docs/images/static_assets/image9.jpg differ diff --git a/docs/images/static_assets/shelf.jpg b/docs/images/static_assets/shelf.jpg new file mode 100644 index 000000000..b08776cd8 Binary files /dev/null and b/docs/images/static_assets/shelf.jpg differ diff --git a/docs/images/static_assets/shelf_edit.jpg b/docs/images/static_assets/shelf_edit.jpg new file mode 100644 index 000000000..8e1ad67af Binary files /dev/null and b/docs/images/static_assets/shelf_edit.jpg differ diff --git a/docs/images/static_assets/shelf_support.jpg b/docs/images/static_assets/shelf_support.jpg new file mode 100644 index 000000000..eba2a84d8 Binary files /dev/null and b/docs/images/static_assets/shelf_support.jpg differ diff --git a/docs/images/static_assets/table.jpg b/docs/images/static_assets/table.jpg new file mode 100644 index 000000000..79e80b323 Binary files /dev/null and b/docs/images/static_assets/table.jpg differ diff --git a/docs/images/static_assets/untitled10.jpg b/docs/images/static_assets/untitled10.jpg new file mode 100644 index 000000000..99a57b48e Binary files /dev/null and b/docs/images/static_assets/untitled10.jpg differ diff --git a/docs/images/static_assets/untitled11.jpg b/docs/images/static_assets/untitled11.jpg new file mode 100644 index 000000000..7dbfce2c1 Binary files /dev/null and b/docs/images/static_assets/untitled11.jpg differ diff --git a/docs/images/static_assets/untitled12.jpg b/docs/images/static_assets/untitled12.jpg new file mode 100644 index 000000000..ba65068eb Binary files /dev/null and b/docs/images/static_assets/untitled12.jpg differ diff --git a/docs/images/static_assets/untitled13.jpg b/docs/images/static_assets/untitled13.jpg new file mode 100644 index 000000000..4a8cdb0bf Binary files /dev/null and b/docs/images/static_assets/untitled13.jpg differ diff --git a/docs/images/static_assets/untitled14.jpg b/docs/images/static_assets/untitled14.jpg new file mode 100644 index 000000000..ef0c11a17 Binary files /dev/null and b/docs/images/static_assets/untitled14.jpg differ diff --git a/docs/images/static_assets/untitled15.jpg b/docs/images/static_assets/untitled15.jpg new file mode 100644 index 000000000..66f215781 Binary files /dev/null and b/docs/images/static_assets/untitled15.jpg differ diff --git a/docs/images/static_assets/untitled16.jpg b/docs/images/static_assets/untitled16.jpg new file mode 100644 index 000000000..6fa6b9a90 Binary files /dev/null and b/docs/images/static_assets/untitled16.jpg differ diff --git a/docs/images/static_assets/untitled8.jpg b/docs/images/static_assets/untitled8.jpg new file mode 100644 index 000000000..d656aebaf Binary files /dev/null and b/docs/images/static_assets/untitled8.jpg differ diff --git a/docs/images/static_assets/untitled9.jpg b/docs/images/static_assets/untitled9.jpg new file mode 100644 index 000000000..7e335f55a Binary files /dev/null and b/docs/images/static_assets/untitled9.jpg differ diff --git a/docs/images/static_assets/vending.jpg b/docs/images/static_assets/vending.jpg new file mode 100644 index 000000000..348b0d4bd Binary files /dev/null and b/docs/images/static_assets/vending.jpg differ diff --git a/infinigen/__init__.py b/infinigen/__init__.py index ba7bb0af6..ab93dd97b 100644 --- a/infinigen/__init__.py +++ b/infinigen/__init__.py @@ -6,7 +6,7 @@ import logging from pathlib import Path -__version__ = "1.7.1" +__version__ = "1.8.0" def repo_root(): diff --git a/infinigen/assets/static_assets/__init__.py b/infinigen/assets/static_assets/__init__.py new file mode 100644 index 000000000..bfc512116 --- /dev/null +++ b/infinigen/assets/static_assets/__init__.py @@ -0,0 +1,7 @@ +from .base import StaticAssetFactory +from .static_category import ( + StaticShelfFactory, + StaticSofaFactory, + StaticTableFactory, + static_category_factory, +) diff --git a/infinigen/assets/static_assets/base.py b/infinigen/assets/static_assets/base.py new file mode 100644 index 000000000..e23cadef2 --- /dev/null +++ b/infinigen/assets/static_assets/base.py @@ -0,0 +1,117 @@ +# Copyright (C) 2024, Princeton University. +# This source code is licensed under the BSD 3-Clause license found in the LICENSE file in the root directory of this source tree. + +# Authors: +# - Karhan Kayan + +import os +from typing import Callable, Dict, Optional + +import bpy + +from infinigen.assets.static_assets.utils import collapse_hierarchy +from infinigen.core.placement.factory import AssetFactory + + +class StaticAssetFactory(AssetFactory): + root_asset_dir = os.path.join(os.path.dirname(os.path.abspath(__file__)), "source") + + import_map: Dict[str, Callable] = { + "dae": bpy.ops.wm.collada_import, + "abc": bpy.ops.wm.alembic_import, + "usd": bpy.ops.wm.usd_import, + "obj": bpy.ops.wm.obj_import, + "ply": bpy.ops.wm.ply_import, + "stl": bpy.ops.wm.stl_import, + "fbx": bpy.ops.import_scene.fbx, + "glb": bpy.ops.import_scene.gltf, + "gltf": bpy.ops.import_scene.gltf, + "blend": bpy.ops.wm.append, + } + + @classmethod + def set_root_asset_dir(cls, directory): + cls.root_asset_dir = directory + + def __init__(self, factory_seed, coarse=False): + super().__init__(factory_seed, coarse) + if self.root_asset_dir is None: + raise ValueError( + "Root asset directory not set. Call StaticAssetFactory.set_root_asset_dir() first." + ) + + def import_single_object_from_blend(self, file_path): + with bpy.data.libraries.load(file_path, link=False) as (data_from, data_to): + # Ensure there is exactly one object + if len(data_from.objects) == 1: + object_name = data_from.objects[0] + elif len(data_from.objects) == 0: + raise ValueError("No objects found in the Blender file.") + else: + raise ValueError("More than one object found in the Blender file.") + + category = "Object" + blendfile = file_path + filepath = os.path.join(blendfile, category, object_name) + filename = object_name + directory = os.path.join(blendfile, category) + + bpy.ops.wm.append(filepath=filepath, filename=filename, directory=directory) + + def import_file(self, file_path: str) -> Optional[bpy.types.Object]: + extension = file_path.split(".")[-1].lower() + if extension in self.import_map: + func = self.import_map.get(extension) + + initial_objects = set(bpy.context.scene.objects) + + if extension in ["glb", "gltf"]: + func(filepath=file_path, merge_vertices=True) + elif extension == "blend": + self.import_single_object_from_blend(file_path) + else: + func(filepath=file_path) + + new_objects = set(bpy.context.scene.objects) - initial_objects + for obj in new_objects: + obj.rotation_mode = "XYZ" + + if new_objects: + # Check if these objects form a tree structure + def is_tree(objects): + roots = [obj for obj in objects if obj.parent is None] + if len(roots) != 1: + return False, None + root = roots[0] + visited = set() + + def dfs(obj): + if obj in visited: + return False + visited.add(obj) + for child in obj.children: + if not dfs(child): + return False + return True + + if dfs(root): + return len(visited) == len(objects), root + else: + return False, None + + is_tree_structure, root_object = is_tree(new_objects) + + if not is_tree_structure: + raise ValueError( + "The imported objects do not form a tree structure." + ) + + collapsed_object = collapse_hierarchy(root_object) + return collapsed_object + else: + return None + else: + raise ValueError(f"Unsupported file format: {extension}") + + def create_asset(self, **params) -> bpy.types.Object: + raise NotImplementedError("This method should be implemented by subclasses") diff --git a/infinigen/assets/static_assets/static_category.py b/infinigen/assets/static_assets/static_category.py new file mode 100644 index 000000000..d5da0beac --- /dev/null +++ b/infinigen/assets/static_assets/static_category.py @@ -0,0 +1,91 @@ +# Copyright (C) 2024, Princeton University. +# This source code is licensed under the BSD 3-Clause license found in the LICENSE file in the root directory of this source tree. + +# Authors: +# - Karhan Kayan + +import os +import random + +import bpy + +from infinigen.assets.static_assets.base import StaticAssetFactory +from infinigen.core.tagging import tag_support_surfaces +from infinigen.core.util.math import FixedSeed + + +def static_category_factory( + category, + tag_support=False, + x_dim: float = None, + y_dim: float = None, + z_dim: float = None, + rotation_euler: tuple[float] = None, +) -> StaticAssetFactory: + """ + Create a factory for external asset import. + tag_support: tag the planes of the object that are parallel to xy plane as support surfaces (e.g. shelves) + x_dim, y_dim, z_dim: specify ONLY ONE dimension for the imported object. The object will be scaled accordingly. + rotation_euler: sets the rotation of the object in euler angles. The object will not be rotated if not specified. + """ + + class StaticCategoryFactory(StaticAssetFactory): + def __init__(self, factory_seed, coarse=False): + super().__init__(factory_seed, coarse) + with FixedSeed(factory_seed): + self.category = category + self.tag_support = tag_support + self.asset_dir = os.path.join(self.root_asset_dir, category) + self.x_dim, self.y_dim, self.z_dim = x_dim, y_dim, z_dim + self.rotation_euler = rotation_euler + asset_files = [ + f + for f in os.listdir(self.asset_dir) + if f.lower().endswith(tuple(self.import_map.keys())) + ] + if not asset_files or len(asset_files) == 0: + raise ValueError(f"No valid asset files found in {self.asset_dir}") + self.asset_file = random.choice(asset_files) + + def create_asset(self, **params) -> bpy.types.Object: + file_path = os.path.join(self.asset_dir, self.asset_file) + imported_obj = self.import_file(file_path) + if ( + self.x_dim is not None + or self.y_dim is not None + or self.z_dim is not None + ): + # check only one dimension is provided + if ( + sum( + [ + 1 + for dim in [self.x_dim, self.y_dim, self.z_dim] + if dim is not None + ] + ) + != 1 + ): + raise ValueError("Only one dimension can be provided") + if self.x_dim is not None: + scale = self.x_dim / imported_obj.dimensions[0] + elif self.y_dim is not None: + scale = self.y_dim / imported_obj.dimensions[1] + else: + scale = self.z_dim / imported_obj.dimensions[2] + imported_obj.scale = (scale, scale, scale) + if self.tag_support: + tag_support_surfaces(imported_obj) + + if imported_obj: + return imported_obj + else: + raise ValueError(f"Failed to import asset: {self.asset_file}") + + return StaticCategoryFactory + + +# Create factory instances for different categories +StaticSofaFactory = static_category_factory("Sofa") +StaticTableFactory = static_category_factory("Table") +StaticShelfFactory = static_category_factory("Shelf", tag_support=True, z_dim=2) diff --git a/infinigen/assets/static_assets/utils.py b/infinigen/assets/static_assets/utils.py new file mode 100644 index 000000000..fb8de8eab --- /dev/null +++ b/infinigen/assets/static_assets/utils.py @@ -0,0 +1,62 @@ +# Copyright (C) 2024, Princeton University. +# This source code is licensed under the BSD 3-Clause license found in the LICENSE file in the root directory of this source tree. + +# Authors: +# - Karhan Kayan +import bpy + +from infinigen.core.util import blender as butil + + +def create_empty_mesh_object(name, parent=None): + mesh = bpy.data.meshes.new(name=f"{name}_mesh") + obj = bpy.data.objects.new(name, mesh) + bpy.context.collection.objects.link(obj) + if parent: + obj.parent = parent + return obj + + +def apply_transforms(obj): + bpy.ops.object.select_all(action="DESELECT") + obj.select_set(True) + bpy.context.view_layer.objects.active = obj + bpy.ops.object.transform_apply(location=True, rotation=True, scale=True) + + +def collapse_hierarchy(root_obj): + mesh_objects = [] + + def process_object(obj, parent=None): + new_obj = obj + if obj.type != "MESH": + new_obj = create_empty_mesh_object(obj.name, parent) + new_obj.matrix_world = obj.matrix_world + else: + if parent: + new_obj.parent = parent + + mesh_objects.append(new_obj) + + for child in obj.children: + process_object(child, new_obj) + + process_object(root_obj) + + # Apply all transformations + for obj in mesh_objects: + apply_transforms(obj) + + # Join all mesh objects + bpy.ops.object.select_all(action="DESELECT") + for obj in mesh_objects: + obj.select_set(True) + + bpy.context.view_layer.objects.active = mesh_objects[0] + bpy.ops.object.join() + + final_obj = bpy.context.active_object + + butil.delete(list(butil.iter_object_tree(root_obj))) + + return final_obj diff --git a/infinigen/core/tagging.py b/infinigen/core/tagging.py index 82347894b..37079f9f3 100644 --- a/infinigen/core/tagging.py +++ b/infinigen/core/tagging.py @@ -2,7 +2,7 @@ # This source code is licensed under the BSD 3-Clause license found in the LICENSE file in the root directory # of this source tree. -# Authors: Yihan Wang, Karhan Kayan: face based tagging, canonical surface tagging, mask extraction +# Authors: Yihan Wang, Karhan Kayan: face based tagging, canonical surface tagging, mask extraction, support tag import json @@ -11,6 +11,7 @@ import bpy import numpy as np +from mathutils import Vector import infinigen.core.util.blender as butil from infinigen.core import surface @@ -451,3 +452,49 @@ def extract_mask( return butil.spawn_vert() return res + + +def tag_support_surfaces(obj, angle_threshold=0.1): + """ + Tags faces of the object (or its mesh children) whose normal is close to the +z direction with the "support" tag. + + Args: + obj (bpy.types.Object): The object to tag (can be any type of object). + angle_threshold (float): The cosine of the maximum angle deviation from +z to be considered a support surface. + """ + + def global_polygon_normal(obj, polygon): + loc, rot, scale = obj.matrix_world.decompose() + rot = rot.to_matrix() + normal = rot @ polygon.normal + return normal / np.linalg.norm(normal) + + def process_mesh(mesh_obj): + up_vector = Vector((0, 0, 1)) + + n_poly = len(mesh_obj.data.polygons) + support_mask = np.zeros(n_poly, dtype=bool) + + for poly in mesh_obj.data.polygons: + global_normal = global_polygon_normal(mesh_obj, poly) + if global_normal.dot(up_vector) > 1 - angle_threshold: + support_mask[poly.index] = True + + if t.Subpart.SupportSurface.value not in tag_system.tag_dict: + tag_system.tag_dict[t.Subpart.SupportSurface.value] = ( + len(tag_system.tag_dict) + 1 + ) + + tag_object(mesh_obj, name=t.Subpart.SupportSurface.value, mask=support_mask) + + print( + f"Tagged {support_mask.sum()} faces as 'support' in object {mesh_obj.name}" + ) + + def process_object(obj): + if obj.type == "MESH": + process_mesh(obj) + for child in obj.children: + process_object(child) + + process_object(obj) diff --git a/infinigen/tools/perceptual/analysis.ipynb b/infinigen/tools/perceptual/analysis.ipynb index 62561058e..929ff2ced 100644 --- a/infinigen/tools/perceptual/analysis.ipynb +++ b/infinigen/tools/perceptual/analysis.ipynb @@ -14,6 +14,7 @@ "import pandas as pd\n", "import statsmodels.api as sm\n", "\n", + "\n", "def A_count_proportion(df):\n", " A_count = 0\n", " B_count = 0\n", @@ -311,7 +312,6 @@ } ], "source": [ - "import statsmodels.api as sm\n", "\n", "def A_count_proportion(dataframes):\n", " A_count = 0\n", diff --git a/infinigen_examples/indoor_asset_semantics.py b/infinigen_examples/indoor_asset_semantics.py index 6f14df158..4d8bb9af8 100644 --- a/infinigen_examples/indoor_asset_semantics.py +++ b/infinigen_examples/indoor_asset_semantics.py @@ -3,6 +3,7 @@ # Authors: Alexander Raistrick +import infinigen.assets.static_assets as static_assets from infinigen.assets.objects import ( appliances, bathroom, @@ -120,6 +121,7 @@ def home_asset_usage(): shelves.SimpleBookcaseFactory, shelves.CellShelfFactory, shelves.LargeShelfFactory, + static_assets.StaticShelfFactory, shelves.KitchenCabinetFactory, shelves.SingleCabinetFactory, } @@ -136,6 +138,7 @@ def home_asset_usage(): tables.TableCocktailFactory, shelves.SimpleDeskFactory, tables.CoffeeTableFactory, + static_assets.StaticTableFactory, }, ) @@ -147,6 +150,7 @@ def home_asset_usage(): used_as[Semantics.LoungeSeating] = { seating.SofaFactory, + static_assets.StaticSofaFactory, seating.ArmChairFactory, } @@ -183,6 +187,7 @@ def home_asset_usage(): bathroom.ToiletFactory, bathroom.BathtubFactory, seating.SofaFactory, + static_assets.StaticSofaFactory, shelves.TVStandFactory, }, ) @@ -280,6 +285,7 @@ def home_asset_usage(): shelves.SingleCabinetFactory, shelves.KitchenCabinetFactory, shelves.LargeShelfFactory, + static_assets.StaticShelfFactory, table_decorations.SinkFactory, tables.TableCocktailFactory, }