Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Text Tower Refactor #185

Merged
merged 28 commits into from
Nov 4, 2022
Merged

Text Tower Refactor #185

Show file tree
Hide file tree
Changes from 19 commits
Commits
Show all changes
28 commits
Select commit Hold shift + click to select a range
4ce3926
POC interface for create fn with mandatory pretrained arg and inferen…
rwightman Sep 17, 2022
1c8844b
Text Tower Support
iejMac Sep 25, 2022
1e348c8
add model.py changes
iejMac Sep 25, 2022
213e218
modified_resnet.py
iejMac Sep 25, 2022
b4df97a
update visual transformer to main version
iejMac Sep 25, 2022
9cf309e
init_params
iejMac Sep 25, 2022
f454a18
update to main
iejMac Sep 25, 2022
f474a2a
import math
iejMac Sep 25, 2022
65a14a9
remove print
iejMac Sep 25, 2022
b3a3718
convert state dict
iejMac Sep 25, 2022
9955855
comment above conver_state_dict
iejMac Sep 25, 2022
2415600
update to main
iejMac Sep 25, 2022
dc9fc01
comment in factory
Sep 25, 2022
15daeb7
Add a note on how to do smaller epochs. Fix #135
rom1504 Sep 24, 2022
a618008
Recommend img2dataset in readme, fix #148
rom1504 Sep 24, 2022
75a009b
filter examples with no images, in addition to those with no captions…
mehdidc Sep 26, 2022
aa71712
Add jit=True to check we don't break torchscript
rom1504 Sep 26, 2022
c849dee
Test both jit True and False
rom1504 Sep 26, 2022
044c30d
Merge branch 'text_tower' of https://github.com/iejMac/open_clip into…
rwightman Sep 29, 2022
2c3d86e
Refactor custom text tower into separate model w/ code re-use reduced.
rwightman Sep 29, 2022
d79c5c2
Remove text tower conversion for OpenAI weight loads
rwightman Oct 31, 2022
3e76bca
Merge remote-tracking branch 'origin/from_pretrained' into text_tower…
rwightman Oct 31, 2022
c4190d2
Fixing float16/bfloat16 (pure) modes, adding flag to force use of cus…
rwightman Nov 1, 2022
90a890f
Add/remove some model configs. Add profiler. Add support for layer_sc…
rwightman Nov 3, 2022
0fd8534
Remove save that was for openai checkpoint tests
rwightman Nov 3, 2022
7e5546d
Fix grad checkpoing for timm models (bug). Change grad clipping arg n…
rwightman Nov 4, 2022
e5a92e2
Tweak profile script, fix a bug for resnet models, add G/e/S-32-alt c…
rwightman Nov 4, 2022
c119c01
Bump version to 2.1.0
rwightman Nov 4, 2022
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
5 changes: 4 additions & 1 deletion src/open_clip/factory.py
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -10,7 +10,7 @@
import torch

from .constants import OPENAI_DATASET_MEAN, OPENAI_DATASET_STD
from .model import CLIP, convert_weights_to_fp16, resize_pos_embed
from .model import CLIP, CustomTextCLIP, convert_weights_to_fp16, convert_to_custom_text_state_dict, resize_pos_embed
from .openai import load_openai_model
from .pretrained import get_pretrained_cfg, download_pretrained
from .transform import image_transform
Expand Down Expand Up @@ -61,6 +61,9 @@ def load_state_dict(checkpoint_path: str, map_location='cpu'):

def load_checkpoint(model, checkpoint_path, strict=True):
state_dict = load_state_dict(checkpoint_path)
# detect old format and make compatible with new format
if 'positional_embedding' in state_dict and not hasattr(model, 'positional_embedding'):
rom1504 marked this conversation as resolved.
Show resolved Hide resolved
state_dict = convert_to_custom_text_state_dict(state_dict)
resize_pos_embed(state_dict, model)
incompatible_keys = model.load_state_dict(state_dict, strict=strict)
return incompatible_keys
Expand Down
Loading