Training on CIFAR10 dataset

[rzyfrank]

I used the code to train CIFAR10 dataset. As mentioned in the paper DDPM, my batchsize is 128, the optimizer is Adam, the learning rate is 0.0002, and I used l2 loss. I found that the training loss kept fluctuating between 0.015 and 0.030. Do I need to reduce the learning rate? Is there a training log that I can refer to?

[jS5t3r]

In the original code it looks like this:

https://github.com/hojonathanho/diffusion/blob/1e0dceb3b3495bbe19116a5e1b3596cd0706c543/scripts/run_cifar.py#L132

    exp_name, tpu_name, bucket_name_prefix, model_name='unet2d16b2', dataset='cifar10',
    optimizer='adam', total_bs=128, grad_clip=1., lr=2e-4, warmup=5000,
    num_diffusion_timesteps=1000, beta_start=0.0001, beta_end=0.02, beta_schedule='linear',
    model_mean_type='eps', model_var_type='fixedlarge', loss_type='mse',
    dropout=0.1, randflip=1,
    tfds_data_dir='tensorflow_datasets', log_dir='logs', keep_checkpoint_max=2