[WIP] update vla ops

[Cathy0908]

No description provided.

[gemini-code-assist]

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request significantly upgrades the video processing capabilities within the system, particularly for tasks involving camera calibration and pose estimation. By introducing new, specialized mappers and refactoring existing ones, it promotes a more modular and efficient workflow. The standardization of camera calibration metadata through CameraCalibrationKeys will ensure consistency and ease of integration across various video operations. These changes aim to improve the accuracy and flexibility of video data preparation for downstream tasks.

Highlights

• New Mapper for Dynamic Camera Intrinsics: A new mapper, VideoCameraCalibrationDroidCalibMapper, has been introduced to dynamically extract camera intrinsics from videos using the DroidCalib method, including automatic dependency management.
• Enhanced Camera Pose Estimation: The existing VideoCameraPoseMapper has been replaced by a new VideoCameraPoseMegaSaMMapper, which leverages MegaSaM and MoGe-2 for more robust and accurate camera pose estimation from video data.
• Improved Modularity for Video Mappers: Several existing video-related mappers (DeepCalib, MoGe, HaWoR, VideoUndistort) have been refactored to enhance modularity. They now accept pre-extracted video frames and camera calibration data as input, reducing redundant processing within individual mappers.
• Standardized Camera Calibration Metadata: A new CameraCalibrationKeys class has been introduced to standardize metadata fields for camera intrinsics, distortion coefficients, and other calibration parameters, ensuring consistency across different video processing operations.
• Flexible Video Splitting: The VideoSplitByDurationMapper has been enhanced with new parameters to control the output field for split videos and to manage legacy text token splitting behavior, offering greater flexibility in data processing workflows.

🧠 New Feature in Public Preview: You can now enable Memory to help Gemini Code Assist learn from your team's feedback. This makes future code reviews more consistent and personalized to your project's style. Click here to enable Memory in your admin console.

Changelog

• data_juicer/ops/mapper/init.py
  • Imported VideoCameraCalibrationDroidCalibMapper.
  • Removed import for VideoCameraPoseMapper.
  • Imported VideoCameraPoseMegaSaMMapper.
  • Added VideoCameraCalibrationDroidCalibMapper to all.
  • Added VideoCameraPoseMegaSaMMapper to all.
• data_juicer/ops/mapper/video_camera_calibration_droidcalib_mapper.py
  • Added new mapper VideoCameraCalibrationDroidCalibMapper to extract camera intrinsics using DroidCalib.
• data_juicer/ops/mapper/video_camera_calibration_static_deepcalib_mapper.py
  • Removed unused imports.
  • Updated init parameters, removing frame extraction related arguments and adding frame_field.
  • Refactored process_single to directly use frames from frame_field and store results in a list under MetaKeys.static_camera_calibration_deepcalib_tags.
• data_juicer/ops/mapper/video_camera_calibration_static_moge_mapper.py
  • Removed unused imports.
  • Updated init parameters, removing frame extraction related arguments and adding frame_field and output control flags.
  • Added _need_anything method to check for missing outputs.
  • Refactored process_single to directly use frames from frame_field and store results in a list under MetaKeys.static_camera_calibration_moge_tags.
• data_juicer/ops/mapper/video_camera_pose_mapper.py
  • Removed the entire file.
• data_juicer/ops/mapper/video_camera_pose_megasam_mapper.py
  • Added new mapper VideoCameraPoseMegaSaMMapper for camera pose estimation using MegaSaM.
• data_juicer/ops/mapper/video_hand_reconstruction_hawor_mapper.py
  • Removed unused imports.
  • Updated init parameters, removing MoGe-related arguments and adding frame_field and camera_calibration_field.
  • Removed internal MoGe-based FoV estimation and frame extraction logic, now relying on external fields.
  • Refactored process_single to use pre-computed FoV and frames, and store results in a list.
• data_juicer/ops/mapper/video_split_by_duration_mapper.py
  • Imported loguru.
  • Added save_field and legacy_split_by_text_token parameters to init.
  • Implemented logic to handle video splitting based on legacy_split_by_text_token and save to save_field.
  • Added movflags="frag_keyframe+empty_moov" to ffmpeg output for better compatibility.
• data_juicer/ops/mapper/video_undistort_mapper.py
  • Removed unused imports.
  • Added get_global_intrinsics helper function.
  • Made output_video_dir a required parameter and added camera_calibration_field.
  • Removed opencv-contrib-python and numpy installation checks.
  • Added movflags="frag_keyframe+empty_moov" to ffmpeg output.
  • Refactored process_single to handle multiple videos per sample and derive intrinsics/xi from camera_calibration_field.
• data_juicer/utils/constant.py
  • Added CameraCalibrationKeys class with various camera calibration metadata fields.
  • Added MetaKeys.camera_calibration_droidcalib_tags.
• data_juicer/utils/model_utils.py
  • Updated error message for prepare_deepcalib_model to explicitly require a CUDA device.
  • Modified wget commands for HaWoR model and config paths to include the -O flag for specifying output filenames.
• docs/Operators.md
  • Updated mapper count from 103 to 104.
  • Added video_camera_calibration_droidcalib_mapper and video_camera_pose_megasam_mapper entries.
  • Removed video_camera_pose_mapper entry.
• tests/ops/mapper/test_video_camera_calibration_static_deepcalib_mapper.py
  • Updated assertions to use CameraCalibrationKeys for accessing metadata.
• tests/ops/mapper/test_video_camera_calibration_static_moge_mapper.py
  • Updated assertions to use CameraCalibrationKeys for accessing metadata.
• tests/ops/mapper/test_video_camera_pose_mapper.py
  • Updated assertions to use CameraCalibrationKeys for accessing metadata.

Activity

• The pull request was created by Cathy0908.
• The title indicates it is a Work In Progress.
• No specific comments or reviews have been made yet.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	/gemini review	Performs a code review for the current pull request in its current state.
Pull Request Summary	/gemini summary	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	/gemini help	Displays a list of available commands.

Customization

To customize Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counter productive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for Github and other Google products, sign up here.

You can also get AI-powered code generation, chat, as well as code reviews directly in the IDE at no cost with the Gemini Code Assist IDE Extension.

Footnotes

1. Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

[gemini-code-assist]

Code Review

This pull request refactors several video processing operators, decoupling frame extraction and improving modularity, with key changes like adding a new VideoCameraCalibrationDroidCalibMapper and renaming VideoCameraPoseMapper. However, significant security concerns exist due to supply chain risks and arbitrary file overwrite vulnerabilities, as one operator downloads code from an unverified personal GitHub repository and others allow arbitrary file overwrites via unvalidated user-provided paths in wget commands. Additionally, critical issues related to error handling, incomplete feature implementation, and dependency management need to be addressed.

[Cathy0908]

/gemini