Release 0.4.6

Fixed a significant bug that happens in some intuitivePhysics scenes with objects that move much more slowly than intended.

Release 0.4.5

This is the Evaluation 4 feature freeze release. The TA2 development team will pause active development on the MCS python package and corresponding Unity run time but will respond to any breaking issues that may arise during TA1 training.

• Config now accepts a dictionary or a file path string with env override (MCS_CONFIG_FILE_PATH) used for evaluation purposes.
• Removed handling of the MCS_METADATA_LEVEL environment variable.
• Renamed ObjectMetadata.color to ObjectMetadata.segment_color to lessen confusion surrounding that attribute. The segment_color is associated with the instance segmentation mask.
• Improved top-down plotter performance, agent heading appearance, and visibility of exterior room walls.
• Fixed a bug in which metallic materials were extremely reflective.
• Hide the performer agent's body.
• Using Open, Close, or Put Object with the floor or walls will now return NOT_INTERACTABLE.
• Added more information to JSON scene schema doc.
• Rearranged scripts folder so entry_point console scripts functions correctly.

Release 0.4.4

Simplified Installation

The machine_common_sense Python library is now available on PyPi and can be installed with "pip install machine_common_sense".

Additionally, if you do not provide a path to the Unity app when creating a new MCS controller, the latest version will automatically be downloaded from our continuous deployment pipeline.

For details, see the Quickstart Installation instructions in our README.

GitHub Documentation

Most of the documentation that was previously hosted in our GitHub repository has now moved to GitHub Pages. Please note that we will continue to use Confluence for additional program-specific documentation.

Frame-by-Frame and End-of-Scene Reporting

We’ve removed the controller.make_step_prediction() function that was previously used to report frame-by-frame scores for passive / VoE scenes. Now, all frame-by-frame scores should be reported using the new report argument in the controller.end_scene() function. Please see our corresponding GitHub documentation for details.

Additionally, we’ve renamed the arguments of the controller.end_scene() function (from choice to rating and confidence to score) to match the terminology used in the Eval 4 Plan.

Finally, we’ve updated our GitHub documentation for the controller.end_scene() function and the passive / VoE goal categories (a.k.a. scene / task types) to describe the expected arguments and formats for each scene type (again, from the Eval 4 Plan).

Please let us know if you have any questions about these changes.

Improved Serialization

We're now using marshmallow to serialize data sent between Python and Unity.

What does this mean for users?

Previously TA1 had access to the "debug" scene files from Eval 3 and 3.5 via our Evaluation Dashboard (UI). (Each "debug" scene file corresponds to a "normal" scene file, but has extra debug properties used by our UI. Each pair of scene files have similar filenames, but the “debug” scene file ends with “_debug.json”) Because not all of the properties in these old "debug" scene files are supported by our new marshmallow schema, you won't be able to run "debug" scene files from Eval 3 and 3.5 using the machine_common_sense Python library from version 0.4.4 forward. You can still run the corresponding "normal" scene files from Eval 3 and 3.5, so you can continue training on that data. Here are ZIPs containing those files:

• https://eval-3-data.s3.amazonaws.com/eval_3_agent_efficient_action_path_lure.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_agent_efficient_action_time_control.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_agent_object_preference.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_interactive_container.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_interactive_obstacle.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_interactive_occluder.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_object_permanence.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_shape_constancy.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_spatio_temporal_continuity.zip
• https://eval-3-data.s3.amazonaws.com/eval_3_5_gravity_support.zip

Unity Addressables

We’ve modernized our Unity codebase by converting to the Addressable Asset System.

What does this mean for users?

Most notably, you’ll notice that the size of our Unity releases is significantly smaller.

Each time you run the MCS simulation environment (v0.4.4+), before loading the scene, it will automatically download and cache the assets (models, textures, etc.) used in that scene, if you don’t already have them cached on your computer. This may cause your run to pause for a few seconds prior to running any scene containing assets that aren’t cached on your computer. If you want to cache all MCS assets on your computer to avoid pauses, please see our instructions for running the cache_addressables script.

Previously, we released instructions for adding your own 3D models into our Unity simulation environment. One limitation is that you have to rebuild the Unity application yourself, and, since TA2 is not legally allowed to distribute the raw files for some of the models used in MCS releases, any self-built Unity applications won’t have access to some of the models. Using addressables will allow us to distribute these models as asset bundles, rather than raw files, and thus remove this limitation. We hope to have this capability available as part of our next release.

Python Logging

We’ve added logging via the Python “logging” package to the machine_common_sense library. Please see our documentation for details.

Release 0.4.3

• Fixes an issue with object (instance segmentation) masks in passive agent instrumental action scenes.
• Added soccer ball model to public GitHub repository.

Release 0.4.2

• The machine_common_sense Python library now supports configurable logging. Please see our README for details.
• The StepMetdata class now contains the performer agent’s reach and the scene’s FPS within the simulation environment (which is currently always set to 20 FPS).
• Most object textures have been compressed in order to reduce the size of the Unity build (from 3 GB to 2 GB). These compressed textures should be visually indistinguishable from their originals.
• We’ve changed both GoalMetadata.action_list and StepMetadata.action_list to return lists of tuples rather than lists of strings. Previously, in Eval 2 and 3, these lists just contained action strings, like ”Pass”. Now, since new tasks in Eval 4 necessitate that these lists have both action strings and parameters, both action_list properties will now use tuples containing action strings and parameters, like (“Pass”, {}). This should be more convenient moving forward since you can pass this data as the arguments to an MCS controller.step function call. Please see our API for details.

Release 0.4.1.1

This build fixes a bug with the camera height in passive scenes.

Also please note that you now need to download the UnityPlayer.so file and keep it in the same directory as the executable file and the Data directory in order to run the simulation environment successfully.

Release 0.4.1

Python and Unity Changelog:

• Updated underlying AI2-THOR Python and Unity library versions from 2.2.0 to 2.5.0
• Added ability to stand on and fall off of platforms
• Added new options to configure room outer dimensions and individual wall materials
• Added optional teleport parameters to EndHabituation action
• Added support and colors for new collision scenes
• Changed default "bounciness" of some objects from 0 to 0.75
• Fixed a bug in which the performer agent's collider was too big
• Removed ffmpeg dependency from Python library
• Updated example scene templates
• Updated SCHEMA documentation with new JSON scene configuration file options

Linux Version:

• MCS-AI2-THOR-Unity-App-v0.4.1.x86_64
• MCS-AI2-THOR-Unity-App-v0.4.1_Data.tar.gz

Mac Version:

• MCS-AI2-THOR-Unity-App-v0.4.1.app.zip

Release 0.4.0

This is the first release of the MCS 3D simulation environment intended for training prior to the summer 2021 MCS evaluation (Eval 4). This release includes the following changes:

• Changed the performer agent's standing height in all interactive scenes to 0.762 (previously 0.4625).
• Added the soccer_ball object type, which will be used as the target object in all interactive scenes.
• Added new example scenes.
• Implemented other minor enhancements, bug fixes, and GitHub documentation updates.

Gravity Support Evaluation 0.3.8 Feature Freeze

The major change in this version is that the ObjectMetdata class now has a state_list attribute that returns zero or more strings describing the object's current state in the scene. This output will be used with the "placer" poles in the gravity support scenes to describe when the "magnetic" / "sticky" mechanism is "active" and when it is "inactive", so you have access to both a visual cue in the RGB image output (blue/pink) and a metadata cue. Please note that the state_list will only be returned running in the oracle metadata tier.

There are also a few other minor enhancements and bug fixes in this version, though most of them aren't really relevant to the gravity support evaluation.

• Initializing scenes with data containing a numpy array will no longer crash the system
• Picked-up objects will now retain their rotation
• Fixed bugs with stacking objects using the PutObject action
• Fixed bugs with dropping held objects
• Fixed bugs with marking objects as visible in the output metadata while holding objects
• Fixed looking up or down too far (past +/- 90 degrees) to correctly do nothing and return a CANNOT_ROTATE action status
• Fixed the status returned on some failed actions to be more sensible (OUT_OF_REACH vs OBSTRUCTED vs NOT_*)
• Fixed "playroom" scene not loading properly

Gravity Support Evaluation 0.3.7

Updates to the Unity 3D simulation environment to facilitate the gravity support evaluation.