v3.1.3 - Kandinsky5, checkpoint providers (S3, Azure, Backblaze, Dropbox), background uploads

New Features

• Kandinsky5 Video & Image training - t2i, t2v, i2v, i2i
• FSDP2 grad norm clipping
• Optional background uploads for checkpoints to avoid blocking training
• Optional external script execution to run validation via cloud or 2nd GPU
• Custom Accelerate tracker support via simpletuner/custom-trackers/ plugin dir

Bugfixes

• Training config wizard now replaces the config correctly
• Qwen Image validations should now work reliably again
• Web UI issue with event lifecycle not showing / clearing prematurely

What's Changed

• (#916) add --push_to_hub_background to publish models asynchronously by @bghira in #2003
• support S3, Backblaze, Azure Blob and Dropbox checkpoint publishing targets by @bghira in #2004
• clean up test outputs and fix minor/sporadic issues by @bghira in #2006
• config wizard should overwrite entire environment it replaces by @bghira in #2007
• add --validation_method which defaults to simpletuner-local; add --validation_external_script for user-provided path and arguments to run validation pipeline with by @bghira in #2008
• (#1730) support custom accelerate trackers by @bghira in #2010
• qwen edit: v2 should not instantiate embed processor; v1 should cache pixel grid and collect captions for later processing during ref-image embedding; ref-images should be embedded instead of target-images by @bghira in #2009
• Revert "qwen edit: v2 should not instantiate embed processor; v1 should cache pixel grid and collect captions for later processing during ref-image embedding; ref-images should be embedded instead of target-images" by @bghira in #2012
• merge by @bghira in #2011
• ace-step: demo config by @bghira in #2014
• qwen-image does not have image parameter on encode_prompt by @bghira in #2015
• training event lifecycle was hidden when training state was running by @bghira in #2016
• qwen-image: validation should pack tensors when input is not by @bghira in #2017
• add problem-solving tips to AGENTS.md by @bghira in #2018
• remove --cache_clear_validation_prompts as currently the prompts are replaced at every startup by @bghira in #2019
• Kandinsky-5 Video and Image model training (T2V, I2V, T2I, I2I) by @bghira in #2013
• FSDP2 support for clip max grad norm by @bghira in #2020
• merge by @bghira in #2021

Full Changelog: v3.1.2...v3.1.3