Transfer Learning Using Diff Pruning

Sources:

• [Diff Pruning 2020 Paper] [Repository]
• [BioBERT 2019 Paper]

Results

Task 1: finetuning BioBERT with diff-vector on BC2GM

• Precision: 85.56
• Recall: 87.02
• F1: 86.28

Parameter	Value
epochs	80
w learning rate	1e-5
alpha learning rate	1e-5
weight decay	1e-2
batch size	32

Task 2: diff pruning BioBERT with diff-vector on BC2GM

Parameter	Value
epochs	80
w learning rate	1e-5
alpha learning rate	1e-1
sparsity penalty	1.25e-7
weight decay	1e-2
batch size	32

• Nonzero parameters: 1.1%
• Precision: 84.73
• Recall: 83.32
• F1: 86.19

Magnitude Pruning and Fixmask Finetuning:

Parameter	Value
epochs	80
w learning rate	1e-5
alpha learning rate	-
sparsity penalty	-
weight decay	1e-2
batch size	32

• Nonzero parameters:0.5%
• Precision: 86.43
• Recall: 84.33
• F1: 85.37