Name	Name	Last commit message	Last commit date
parent directory ..
criteo1tb	criteo1tb
criteo1tb_embed_init	criteo1tb_embed_init
criteo1tb_layernorm	criteo1tb_layernorm
criteo1tb_resnet	criteo1tb_resnet
fastmri	fastmri
fastmri_layernorm	fastmri_layernorm
fastmri_model_size	fastmri_model_size
fastmri_tanh	fastmri_tanh
fineweb_edu_lm	fineweb_edu_lm
imagenet_resnet	imagenet_resnet
imagenet_resnet_gelu	imagenet_resnet_gelu
imagenet_resnet_large_bn_init	imagenet_resnet_large_bn_init
imagenet_resnet_silu	imagenet_resnet_silu
imagenet_vit	imagenet_vit
imagenet_vit_glu	imagenet_vit_glu
imagenet_vit_map	imagenet_vit_map
imagenet_vit_post_ln	imagenet_vit_post_ln
librispeech_conformer	librispeech_conformer
librispeech_conformer_attention_temperature	librispeech_conformer_attention_temperature
librispeech_conformer_gelu	librispeech_conformer_gelu
librispeech_conformer_layernorm	librispeech_conformer_layernorm
librispeech_deepspeech	librispeech_deepspeech
librispeech_deepspeech_no_resnet	librispeech_deepspeech_no_resnet
librispeech_deepspeech_norm_and_spec_aug	librispeech_deepspeech_norm_and_spec_aug
librispeech_deepspeech_tanh	librispeech_deepspeech_tanh
ogbg	ogbg
ogbg_gelu	ogbg_gelu
ogbg_model_size	ogbg_model_size
ogbg_silu	ogbg_silu
wmt	wmt
wmt_attention_temp	wmt_attention_temp
wmt_glu_tanh	wmt_glu_tanh
wmt_post_ln	wmt_post_ln
README.md	README.md
__init__.py	__init__.py
cosine_warmup.py	cosine_warmup.py
data_selection.py	data_selection.py
get_batch_size.py	get_batch_size.py
jax_adamw.py	jax_adamw.py
jax_momentum.py	jax_momentum.py
jax_nadamw.py	jax_nadamw.py
jax_nesterov.py	jax_nesterov.py
jax_submission_base.py	jax_submission_base.py
pytorch_adamw.py	pytorch_adamw.py
pytorch_momentum.py	pytorch_momentum.py
pytorch_nadamw.py	pytorch_nadamw.py
pytorch_nesterov.py	pytorch_nesterov.py
pytorch_submission_base.py	pytorch_submission_base.py

Target Setting Run replications

Original runs were run on Google TPUv2-8 machines.

These are not valid submissions, because they use a different hyperparameter setting per workload. But we include them in order to reproduce how we set the target metric values.

To simplify directory setting, set:

ROOT_DIR=/home/znado

Criteo

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=criteo1tb \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/criteo1tb/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=criteo1tb \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/criteo1tb/tuning_search_space.json

FastMRI

Target was set using Nesterov with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=fastmri \
    --submission_path=algorithms/target_setting_algorithms/jax_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/fastmri/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=fastmri \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/fastmri/tuning_search_space.json

ImageNet-Resnet

Target was set using Heavy-ball Momentum with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_resnet \
    --submission_path=algorithms/target_setting_algorithms/jax_momentum.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_resnet/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/imagenet_pytorch \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_resnet \
    --submission_path=algorithms/target_setting_algorithms/pytorch_momentum.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_resnet/tuning_search_space.json

ImageNet-ViT

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_vit \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_vit/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/imagenet_pytorch \
    --imagenet_v2_data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=imagenet_vit \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/imagenet_vit/tuning_search_space.json

Librispeech-Conformer

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_conformer \
    --submission_path=algorithms/target_setting_algorithms/jax_adamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_conformer/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_conformer \
    --submission_path=algorithms/target_setting_algorithms/pytorch_adamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_conformer/tuning_search_space.json

Librispeech-Deepspeech

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_deepspeech \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_deepspeech/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=librispeech_deepspeech \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/librispeech_deepspeech/tuning_search_space.json

OGBG

Target was set using Nesterov with a linear warmup and linear decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=ogbg \
    --submission_path=algorithms/target_setting_algorithms/jax_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/ogbg/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=ogbg \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nesterov.py \
    --tuning_search_space=algorithms/target_setting_algorithms/ogbg/tuning_search_space.json

WMT

Target was set using NAdamW with a linear warmup cosine decay LR schedule.

python3 submission_runner.py \
    --framework=jax \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=wmt \
    --submission_path=algorithms/target_setting_algorithms/jax_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/wmt/tuning_search_space.json

torchrun --redirects 1:0,2:0,3:0,4:0,5:0,6:0,7:0 --standalone --nnodes=1 --nproc_per_node=8 submission_runner.py \
    --framework=pytorch \
    --data_dir=$ROOT_DIR/tensorflow_datasets \
    --experiment_dir=$ROOT_DIR \
    --experiment_name=target_setting \
    --workload=wmt \
    --submission_path=algorithms/target_setting_algorithms/pytorch_nadamw.py \
    --tuning_search_space=algorithms/target_setting_algorithms/wmt/tuning_search_space.json

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

README.md

Target Setting Run replications

Criteo

FastMRI

ImageNet-Resnet

ImageNet-ViT

Librispeech-Conformer

Librispeech-Deepspeech

OGBG

WMT

FilesExpand file tree

target_setting_algorithms

Directory actions

More options

Directory actions

More options

Latest commit

History

target_setting_algorithms

Folders and files

parent directory

README.md

Target Setting Run replications

Criteo

FastMRI

ImageNet-Resnet

ImageNet-ViT

Librispeech-Conformer

Librispeech-Deepspeech

OGBG

WMT