Code

adding CoCa#256

Code

Closed

gpucce wants to merge 123 commits intomlfoundations:mainmlfoundations/open_clip:mainfrom gpucce:add_cocagpucce/open_clip:add_cocaCopy head branch name to clipboard

+553-42

Commits

Commits on Nov 25, 2022

• initial setup

  gpucce

  committed
  1189487

  Copy full SHA for 1189487

Commits on Nov 27, 2022

• add coca loss

  gpucce

  committed
  91d01fa

  Copy full SHA for 91d01fa

• remove loss from the model

  gpucce

  committed
  efb6540

  Copy full SHA for efb6540

• fix loss

  gpucce

  committed
  669a3a0

  Copy full SHA for 669a3a0

• add underscores

  gpucce

  committed
  f081dc4

  Copy full SHA for f081dc4

• name changes

  gpucce

  committed
  0b1c895

  Copy full SHA for 0b1c895

• Merge remote-tracking branch 'upstream/main' into add_coca

  gpucce

  committed
  27369b6

  Copy full SHA for 27369b6

Commits on Nov 29, 2022

• add cross attention to Residual and CustomResidual

  gpucce

  committed
  d518dd0

  Copy full SHA for d518dd0

• fix if

  gpucce

  committed
  11bf57c

  Copy full SHA for 11bf57c

• ädd transformer 'decoder'

  gpucce

  committed
  f3dedf6

  Copy full SHA for f3dedf6

• minor fix

  gpucce

  committed
  50c4726

  Copy full SHA for 50c4726

• looks better

  gpucce

  committed
  1e41d83

  Copy full SHA for 1e41d83

• initlize coca model structure

  gpucce

  committed
  0d91609

  Copy full SHA for 0d91609

• clean

  gpucce

  committed
  50e0cbe

  Copy full SHA for 50e0cbe

• typo and format

  gpucce

  committed
  93b4236

  Copy full SHA for 93b4236

• checkpoint signature

  gpucce

  committed
  97e3c0f

  Copy full SHA for 97e3c0f

Commits on Nov 30, 2022

• adjust multimodal decoder and add CoCaTransformer

  gpucce

  committed
  6ae6f8c

  Copy full SHA for 6ae6f8c

Commits on Dec 1, 2022

• keep older logic

  gpucce

  committed
  0975dfe

  Copy full SHA for 0975dfe

• remove chunk

  gpucce

  committed
  f2265ec

  Copy full SHA for f2265ec

• typo

  gpucce

  committed
  9d47f0e

  Copy full SHA for 9d47f0e

• fix

  gpucce

  committed
  6a101ec

  Copy full SHA for 6a101ec

• make chunk dim explicit

  gpucce

  committed
  e259851

  Copy full SHA for e259851

• adjust cfg names

  gpucce

  committed
  7fff61d

  Copy full SHA for 7fff61d

• add attentionalpooling

  gpucce

  committed
  abd132d

  Copy full SHA for abd132d

• add attentional pooling to coca

  gpucce

  committed
  452d7d2

  Copy full SHA for 452d7d2

• small change

  gpucce

  committed
  43ce18f

  Copy full SHA for 43ce18f

Commits on Dec 2, 2022

• add cocatransformer variants and AttentionPooling

  gpucce

  committed
  3f0f012

  Copy full SHA for 3f0f012

• remoive older attention pooler

  gpucce

  committed
  3e745ec

  Copy full SHA for 3e745ec

• adapt embed text to coca text transformer

  gpucce

  committed
  4f4d3b7

  Copy full SHA for 4f4d3b7

• rm coca layers

  gpucce

  committed
  42539aa

  Copy full SHA for 42539aa

Commits on Dec 3, 2022

• rename and remove useless CoCa models

  gpucce

  committed
  914a570

  Copy full SHA for 914a570

• make attentionpooler pooler only

  gpucce

  committed
  6215d4a

  Copy full SHA for 6215d4a

Commits on Dec 5, 2022

• refactor for one transformer only

  gpucce

  committed
  b97db74

  Copy full SHA for b97db74

• coca forward works

  

  gpucce

  committed
  d89f018

  Copy full SHA for d89f018

• separatae context and n_queries

  

  gpucce

  committed
  9a8c15d

  Copy full SHA for 9a8c15d

• add inital coca_base config

  

  gpucce

  committed
  c8b9236

  Copy full SHA for c8b9236

• remove config

  

  gpucce

  committed
  d0f995a

  Copy full SHA for d0f995a

• small loss change

  

  gpucce

  committed
  5260774

  Copy full SHA for 5260774

• init training file

  

  gpucce

  committed
  7a2b84e

  Copy full SHA for 7a2b84e

• make variable order right

  gpucce

  committed
  3ef1d17

  Copy full SHA for 3ef1d17

• remove print

  gpucce

  committed
  86f47bb

  Copy full SHA for 86f47bb

• uniform names

  gpucce

  committed
  c6834b5

  Copy full SHA for c6834b5

• renaming

  gpucce

  committed
  7489c68

  Copy full SHA for 7489c68

• add coca funcs to init

  gpucce

  committed
  59503df

  Copy full SHA for 59503df

• add coca config and exclude from testing

  gpucce

  committed
  504febd

  Copy full SHA for 504febd

• add and comment simple test (no trained model)

  gpucce

  committed
  72a7e96

  Copy full SHA for 72a7e96

Commits on Dec 6, 2022

• add L2 norm

  

  gpucce

  committed
  d8a94be

  Copy full SHA for d8a94be

• make L2 same as in clip

  

  gpucce

  committed
  d250eac

  Copy full SHA for d250eac

• remove unused temperature

  

  gpucce

  committed
  8d9dfa6

  Copy full SHA for 8d9dfa6

• type

  

  gpucce

  committed
  1f2578c

  Copy full SHA for 1f2578c

• clean

  

  gpucce

  committed
  d8ff1bd

  Copy full SHA for d8ff1bd

• fix config

  

  gpucce

  committed
  fa24047

  Copy full SHA for fa24047

• make rename and move cfg

  

  gpucce

  committed
  f61f9d5

  Copy full SHA for f61f9d5

• rename

  

  gpucce

  committed
  4b76187

  Copy full SHA for 4b76187

• temptative add coca to factory

  

  gpucce

  committed
  b8777fe

  Copy full SHA for b8777fe

• fix config

  

  gpucce

  committed
  42aa408

  Copy full SHA for 42aa408

• update config

  

  gpucce

  committed
  1044f36

  Copy full SHA for 1044f36

Commits on Dec 7, 2022

• embed contrastive cls token in model

  

  gpucce

  committed
  dab7d7d

  Copy full SHA for dab7d7d

• remove unused arg

  

  gpucce

  committed
  d0ae683

  Copy full SHA for d0ae683

• import create_loss

  

  gpucce

  committed
  5a40804

  Copy full SHA for 5a40804

• make factory accept coca

  

  gpucce

  committed
  6789438

  Copy full SHA for 6789438

• make caption loss distributed

  

  gpucce

  committed
  60865ef

  Copy full SHA for 60865ef

• make loss customizable

  

  gpucce

  committed
  ac617bf

  Copy full SHA for ac617bf

• pass loss trhough training_epoch

  

  gpucce

  committed
  b9c2b25

  Copy full SHA for b9c2b25

• add coca specific params to params

  

  gpucce

  committed
  ccfd1e4

  Copy full SHA for ccfd1e4

• removed decoder unused parameters

  

  gpucce

  committed
  c1556d4

  Copy full SHA for c1556d4

• remove unused attributes

  

  gpucce

  committed
  68d608a

  Copy full SHA for 68d608a

• adjust coca_config

  

  gpucce

  committed
  59d4db4

  Copy full SHA for 59d4db4

• Merge remote-tracking branch 'upstream/main' into add_coca

  

  gpucce

  committed
  4ee12e1

  Copy full SHA for 4ee12e1

• fix config and remove unused parameters

  

  gpucce

  committed
  732f15f

  Copy full SHA for 732f15f

• remove comment

  

  gpucce

  committed
  17072c6

  Copy full SHA for 17072c6

• remove more comments

  gpucce

  committed
  74d5e37

  Copy full SHA for 74d5e37

Commits on Dec 8, 2022

• rename attention pooler

  

  gpucce

  committed
  578aadf

  Copy full SHA for 578aadf

• rename TransformerDecoder

  

  gpucce

  committed
  08f43a3

  Copy full SHA for 08f43a3

• make AttentionalPooler clearer

  

  gpucce

  committed
  812a8bb

  Copy full SHA for 812a8bb

• add local loss logic to cocaloss

  

  gpucce

  committed
  f69f4e0

  Copy full SHA for f69f4e0

• only create loss if train in data

  

  gpucce

  committed
  3c02aa5

  Copy full SHA for 3c02aa5

• remove wrong file

  

  gpucce

  committed
  979cef4

  Copy full SHA for 979cef4

• fix attentional pooler call

  

  gpucce

  committed
  2ec204b

  Copy full SHA for 2ec204b

• not ready for testing

  

  gpucce

  committed
  29c7dfa

  Copy full SHA for 29c7dfa

• really not ready for testing

  

  gpucce

  committed
  5a4126b

  Copy full SHA for 5a4126b

• eof lien

  

  gpucce

  committed
  6e49474

  Copy full SHA for 6e49474

Commits on Dec 9, 2022

• Merge remote-tracking branch 'upstream/main' into add_coca

  

  gpucce

  committed
  288ddf3

  Copy full SHA for 288ddf3

• uniform names

  

  gpucce

  committed
  599d448

  Copy full SHA for 599d448

• add possible generative loss to evaluate

  

  gpucce

  committed
  d7953da

  Copy full SHA for d7953da

• change _build function names

  

  gpucce

  committed
  e2042d4

  Copy full SHA for e2042d4

• remove wrong import

  

  gpucce

  committed
  15c69f8

  Copy full SHA for 15c69f8

• remove local_loss from captioning loss

  

  gpucce

  committed
  c219381

  Copy full SHA for c219381

• Merge branch 'main' into add_coca

  rom1504

  authored
  360408e

  Copy full SHA for 360408e

• indexing error

  

  gpucce

  committed
  5c77e4d

  Copy full SHA for 5c77e4d

• finish renaming

  

  gpucce

  committed
  3f095a6

  Copy full SHA for 3f095a6

• adjust configs

  

  gpucce

  committed
  60f35f3

  Copy full SHA for 60f35f3

• add training test for coca

  rom1504

  authored
  a53f477

  Copy full SHA for a53f477

• simplify captioning loss

  

  gpucce

  committed
  b3f3d68

  Copy full SHA for b3f3d68

• remove hf

  

  gpucce

  committed
  8eb4772

  Copy full SHA for 8eb4772

• fix evaluate and loss

  

  gpucce

  committed
  cf0f857

  Copy full SHA for cf0f857

• remove print

  

  gpucce

  committed
  d547017

  Copy full SHA for d547017

• move projection

  

  gpucce

  committed
  75be611

  Copy full SHA for 75be611

• add coca vit 32 config

  

  gpucce

  committed
  356fb7d

  Copy full SHA for 356fb7d

• test on new config

  

  gpucce

  committed
  8008f25

  Copy full SHA for 8008f25

• adjust coca_base config

  

  gpucce

  committed
  5b54a4b

  Copy full SHA for 5b54a4b

Commits on Dec 10, 2022

• Merge branch 'main' into add_coca

  rom1504

  authored
  292fa6e

  Copy full SHA for 292fa6e

• remove coca from test_inference

  gpucce

  committed
  720dabf

  Copy full SHA for 720dabf

• maybe fix regression test

  gpucce

  committed
  bcb82c4

  Copy full SHA for bcb82c4

• make logits and labels contiguous

  gpucce

  committed
  d0f4947

  Copy full SHA for d0f4947

• simpler logic

  gpucce

  committed
  39f20e6

  Copy full SHA for 39f20e6

• make contiguous after transpose

  gpucce

  committed
  2dde78d

  Copy full SHA for 2dde78d

• last test

  gpucce

  committed
  de4c063

  Copy full SHA for de4c063

Commits on Dec 12, 2022

• try fix loss

  

  gpucce

  committed
  00aa464

  Copy full SHA for 00aa464

• Merge remote-tracking branch 'upstream/main' into add_coca

  

  gpucce

  committed
  b7bea09

  Copy full SHA for b7bea09

Commits on Dec 17, 2022

• CoCa PR: loss fix + rename file

  

  iejmac

  committed
  27bfc7d

  Copy full SHA for 27bfc7d

• wait for feedback on this

  

  iejmac

  committed
  e694999

  Copy full SHA for e694999

• cleanup

  

  iejmac

  committed
  5427b0a

  Copy full SHA for 5427b0a

• Merge pull request #1 from iejMac/add_coca2

  Show description for cc6d13f
  gpucce

  authored
  cc6d13f

  Copy full SHA for cc6d13f

• CoCa PR: add set_grad_checkpointing + fix checkpoint API

  

  iejmac

  committed
  abd7849

  Copy full SHA for abd7849

• Merge pull request #2 from iejMac/grad_checkpointing

  Show description for 19300ad
  gpucce

  authored
  19300ad

  Copy full SHA for 19300ad

Commits on Dec 18, 2022

• CoCa PR: fix eval (which uses encode_x instead of forward)

  

  iejmac

  committed
  919f5a0

  Copy full SHA for 919f5a0

• move making space for CLS token into encode_text

  

  iejmac

  committed
  5b29ec0

  Copy full SHA for 5b29ec0

• rever zs changes + fix

  

  iejmac

  committed
  752de0a

  Copy full SHA for 752de0a

• Merge pull request #3 from iejMac/eval_fix

  Show description for 1360fcd
  gpucce

  authored
  1360fcd

  Copy full SHA for 1360fcd

• Merge remote-tracking branch 'upstream/main' into add_coca

  gpucce

  committed
  cd91d32

  Copy full SHA for cd91d32

Commits on Dec 20, 2022

• add cls mask for pad ids

  

  gpucce

  committed
  64c33d8

  Copy full SHA for 64c33d8

• simplify encode image

  

  gpucce

  committed
  17813eb

  Copy full SHA for 17813eb