In some case. there need some padding for divisible by 128. For the padding block, should we choose all padding blocks, or just use decay logic to choose some of them.
The default code choose all padding blocks. Is there an exp to support this is better way?
|
final_log_mask[video_text_border:] = True |
|
final_log_mask[:, video_text_border:] = True |
In some case. there need some padding for divisible by 128. For the padding block, should we choose all padding blocks, or just use decay logic to choose some of them.
The default code choose all padding blocks. Is there an exp to support this is better way?
radial-attention/radial_attn/attn_mask.py
Lines 144 to 145 in 72788d4