Finetuning res101 model pretrained models error

[jsuit]

So if you try and retrain the resnet101 (faster_rcnn_1_10_9771.pth) you get the following error:

   File "trainval_net.py", line 336, in <module>
      optimizer.step()
   File "... pytorch_0.4/lib/python3.6/site-packages/torch/optim/sgd.py", line 101, in step
      buf.mul_(momentum).add_(1 - dampening, d_p)
   RuntimeError: The expanded size of the tensor (512) must match the existing size (1024) at 
  non-singleton dimension 1

It's the optimizer version knowledge of the parameters is different than what it actually is. Also, you get the same error if you use, /home/jonathan/faster-rcnn.pytorch/faster_rcnn_1_6_9771.pth

[EMCP]

this was fixed a while ago I believe.. I've successfully retrained res101 now

#515

[AlexanderHustinx]

Can you include your command-line argument and the full stack-trace?
Might be able to help you out if I can recreate the issue.

Also, what version of Python, PyTorch, CUDA, are you using?

[jsuit]

pytorch 0.4
cuda = 9.0
python 3.6.6
faster_rcnn_1_10_9771.pth (gotten from link provided on README page)

Stack trace:

Preparing training data...
done
loading annotations into memory...
Done (t=2.96s)
creating index...
index created!
before filtering, there are 236574 images...
after filtering, there are 234532 images...
234532 roidb entries
Loading pretrained data/pretrained_model/resnet101_caffe.pth
loading checkpoint faster-rcnn.pytorch/faster_rcnn_1_10_9771.pth
loaded checkpoint faster-rcnn.pytorch/faster_rcnn_1_10_9771.pth
Traceback (most recent call last):
File "trainval_net.py", line 336, in
optimizer.step()
File python3.6/site-packages/torch/optim/sgd.py", line 101, in step
buf.mul_(momentum).add_(1 - dampening, d_p)
RuntimeError: The expanded size of the tensor (512) must match the existing size (1024) at non-singleton dimension 1

[AlexanderHustinx]

Could you also include the command-line argument?
i.e. what you enter in your terminal to run the code

For example:
python demo.py --dataset pascal --net res101 --checksession 1 --checkepoch 7 --checkpoint 10021 --cuda

[jsuit]

   python trainval_net.py --r True --cuda --bs 1

I'm basically saying resume training for the load_name:

(basically, I just changed lines 275-276 in trainval_net.py to
load_name = my path to faster_rcnn_1_10_9771.pth)

Here's the only part I changed. Everything else is the same as the main branch.
if args.resume:
load_name = "path to faster_rcnn_1_10_9771.pth"
print("loading checkpoint %s" % (load_name))
checkpoint = torch.load(load_name)
args.session = checkpoint['session']
args.start_epoch = checkpoint['epoch']
fasterRCNN.load_state_dict(checkpoint['model'])
optimizer.load_state_dict(checkpoint['optimizer'])

my default network is res101 and I'm training on coco2014.

[AlexanderHustinx]

Could you try to add the following tag to your command:
--dataset coco
I think you might be loading the pascal_voc dataset by default.

Additionally, you could try using all the command-line parameters:
python trainval_net.py --dataset coco --net res101 --r True --checksession 1 --checkepoch 10 --checkpoint 9771 --bs 1 --cuda
It is possible you might be overlooking something in your adjusted code.

[zychen2016]

have you solved this problem ? @jsuit

[zychen2016]

Hello, @AlexanderHustinx , I have the same problem

Using res101 pretrained model with trainval_net.py.

But using the same model with test_net.py and demo.py , there is no problem

Using pytorch-1.0 branch

[AlexanderHustinx]

Could you show the stack trace and error?
Maybe I can help spot the issue

[zychen2016]

Could you show the stack trace and error?
Maybe I can help spot the issue

Thanks. @AlexanderHustinx

Using CUDA_VISBLE_DEVICES=0 python3 trainval_net.py --r True --dataset pascal_voc --net vgg16 --bs 1 --nw 0 --lr 1e-3 --lr_decay_step 5 --cuda --checksession 1 --checkepoch 6 --checkpoint 10021 --use_tfb

Called with args:
Namespace(batch_size=1, checkepoch=6, checkpoint=10021, checkpoint_interval=10000, checksession=1, class_agnostic=False, cuda=True, dataset='pascal_voc', disp_interval=100, large_scale=False, lr=0.001, lr_decay_gamma=0.1, lr_decay_step=5, mGPUs=False, max_epochs=20, net='vgg16', num_workers=0, optimizer='sgd', resume=True, save_dir='models', session=1, start_epoch=1, use_tfboard=True)
/data2/CZY/data/faster-rcnn.pytorch/lib/model/utils/config.py:374: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
yaml_cfg = edict(yaml.load(f))
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [8, 16, 32],
'CROP_RESIZE_WITH_MAX_POOL': False,
'CUDA': False,
'DATA_DIR': '/data2/CZY/data/faster-rcnn.pytorch/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'vgg16',
'FEAT_STRIDE': [16],
'GPU_ID': 0,
'MATLAB': 'matlab',
'MAX_NUM_GT_BOXES': 20,
'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
'FIXED_LAYERS': 5,
'REGU_DEPTH': False,
'WEIGHT_DECAY': 4e-05},
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'POOLING_MODE': 'align',
'POOLING_SIZE': 7,
'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '/data2/CZY/data/faster-rcnn.pytorch',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'MODE': 'nms',
'NMS': 0.3,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': False,
'BATCH_SIZE': 256,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'BN_TRAIN': False,
'DISPLAY': 10,
'DOUBLE_BIAS': True,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.01,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
'STEPSIZE': [30000],
'SUMMARY_INTERVAL': 180,
'TRIM_HEIGHT': 600,
'TRIM_WIDTH': 600,
'TRUNCATED': False,
'USE_ALL_GT': True,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0005},
'USE_GPU_NMS': True}
Loaded dataset voc_2007_trainval for training
Set proposal method: gt
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /data2/CZY/data/faster-rcnn.pytorch/data/cache/voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
before filtering, there are 10022 images...
after filtering, there are 10022 images...
10022 roidb entries
Loading pretrained weights from data/pretrained_model/vgg16_caffe.pth
loading checkpoint models/vgg16/pascal_voc/faster_rcnn_1_6_10021.pth
loaded checkpoint models/vgg16/pascal_voc/faster_rcnn_1_6_10021.pth
Traceback (most recent call last):
File "trainval_net.py", line 332, in
optimizer.step()
File "/data2/CZY/software/anconda2/envs/python36/lib/python3.6/site-packages/torch/optim/sgd.py", line 101, in step
buf.mul_(momentum).add_(1 - dampening, d_p)
RuntimeError: The size of tensor a (4096) must match the size of tensor b (3) at non-singleton dimension 3

Using CUDA_VISBLE_DEVICES=0 python3 trainval_net.py --r True --dataset pascal_voc --net res101 --bs 1 --nw 0 --lr 1e-3 --lr_decay_step 5 --cuda --checksession 1 --checkepoch 7 --checkpoint 10021 --use_tfb

Called with args:
Namespace(batch_size=1, checkepoch=7, checkpoint=10021, checkpoint_interval=10000, checksession=1, class_agnostic=False, cuda=True, dataset='pascal_voc', disp_interval=100, large_scale=False, lr=0.001, lr_decay_gamma=0.1, lr_decay_step=5, mGPUs=False, max_epochs=20, net='res101', num_workers=0, optimizer='sgd', resume=True, save_dir='models', session=1, start_epoch=1, use_tfboard=True)
/data2/CZY/data/faster-rcnn.pytorch/lib/model/utils/config.py:374: YAMLLoadWarning: calling yaml.load() without Loader=... is deprecated, as the default Loader is unsafe. Please read https://msg.pyyaml.org/load for full details.
yaml_cfg = edict(yaml.load(f))
Using config:
{'ANCHOR_RATIOS': [0.5, 1, 2],
'ANCHOR_SCALES': [8, 16, 32],
'CROP_RESIZE_WITH_MAX_POOL': False,
'CUDA': False,
'DATA_DIR': '/data2/CZY/data/faster-rcnn.pytorch/data',
'DEDUP_BOXES': 0.0625,
'EPS': 1e-14,
'EXP_DIR': 'res101',
'FEAT_STRIDE': [16],
'GPU_ID': 0,
'MATLAB': 'matlab',
'MAX_NUM_GT_BOXES': 20,
'MOBILENET': {'DEPTH_MULTIPLIER': 1.0,
'FIXED_LAYERS': 5,
'REGU_DEPTH': False,
'WEIGHT_DECAY': 4e-05},
'PIXEL_MEANS': array([[[102.9801, 115.9465, 122.7717]]]),
'POOLING_MODE': 'align',
'POOLING_SIZE': 7,
'RESNET': {'FIXED_BLOCKS': 1, 'MAX_POOL': False},
'RNG_SEED': 3,
'ROOT_DIR': '/data2/CZY/data/faster-rcnn.pytorch',
'TEST': {'BBOX_REG': True,
'HAS_RPN': True,
'MAX_SIZE': 1000,
'MODE': 'nms',
'NMS': 0.3,
'PROPOSAL_METHOD': 'gt',
'RPN_MIN_SIZE': 16,
'RPN_NMS_THRESH': 0.7,
'RPN_POST_NMS_TOP_N': 300,
'RPN_PRE_NMS_TOP_N': 6000,
'RPN_TOP_N': 5000,
'SCALES': [600],
'SVM': False},
'TRAIN': {'ASPECT_GROUPING': False,
'BATCH_SIZE': 128,
'BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'BBOX_NORMALIZE_MEANS': [0.0, 0.0, 0.0, 0.0],
'BBOX_NORMALIZE_STDS': [0.1, 0.1, 0.2, 0.2],
'BBOX_NORMALIZE_TARGETS': True,
'BBOX_NORMALIZE_TARGETS_PRECOMPUTED': True,
'BBOX_REG': True,
'BBOX_THRESH': 0.5,
'BG_THRESH_HI': 0.5,
'BG_THRESH_LO': 0.0,
'BIAS_DECAY': False,
'BN_TRAIN': False,
'DISPLAY': 20,
'DOUBLE_BIAS': False,
'FG_FRACTION': 0.25,
'FG_THRESH': 0.5,
'GAMMA': 0.1,
'HAS_RPN': True,
'IMS_PER_BATCH': 1,
'LEARNING_RATE': 0.001,
'MAX_SIZE': 1000,
'MOMENTUM': 0.9,
'PROPOSAL_METHOD': 'gt',
'RPN_BATCHSIZE': 256,
'RPN_BBOX_INSIDE_WEIGHTS': [1.0, 1.0, 1.0, 1.0],
'RPN_CLOBBER_POSITIVES': False,
'RPN_FG_FRACTION': 0.5,
'RPN_MIN_SIZE': 8,
'RPN_NEGATIVE_OVERLAP': 0.3,
'RPN_NMS_THRESH': 0.7,
'RPN_POSITIVE_OVERLAP': 0.7,
'RPN_POSITIVE_WEIGHT': -1.0,
'RPN_POST_NMS_TOP_N': 2000,
'RPN_PRE_NMS_TOP_N': 12000,
'SCALES': [600],
'SNAPSHOT_ITERS': 5000,
'SNAPSHOT_KEPT': 3,
'SNAPSHOT_PREFIX': 'res101_faster_rcnn',
'STEPSIZE': [30000],
'SUMMARY_INTERVAL': 180,
'TRIM_HEIGHT': 600,
'TRIM_WIDTH': 600,
'TRUNCATED': False,
'USE_ALL_GT': True,
'USE_FLIPPED': True,
'USE_GT': False,
'WEIGHT_DECAY': 0.0001},
'USE_GPU_NMS': True}
Loaded dataset voc_2007_trainval for training
Set proposal method: gt
Appending horizontally-flipped training examples...
voc_2007_trainval gt roidb loaded from /data2/CZY/data/faster-rcnn.pytorch/data/cache/voc_2007_trainval_gt_roidb.pkl
done
Preparing training data...
done
before filtering, there are 10022 images...
after filtering, there are 10022 images...
10022 roidb entries
Loading pretrained weights from data/pretrained_model/resnet101_caffe.pth
loading checkpoint models/res101/pascal_voc/faster_rcnn_1_7_10021.pth
loaded checkpoint models/res101/pascal_voc/faster_rcnn_1_7_10021.pth
Traceback (most recent call last):
File "trainval_net.py", line 332, in
optimizer.step()
File "/data2/CZY/software/anconda2/envs/python36/lib/python3.6/site-packages/torch/optim/sgd.py", line 101, in step
buf.mul_(momentum).add_(1 - dampening, d_p)
RuntimeError: The size of tensor a (512) must match the size of tensor b (1024) at non-singleton dimension 1

3. However , In test_net.py and demo.py , it's Ok.

[AlexanderHustinx]

Oef, okay this has been a while for me 😅

So you can successfully run the test_net.py and demo.py.
Can you run a training session from scratch? (Not resuming)

[zychen2016]

Oef, okay this has been a while for me

So you can successfully run the test_net.py and demo.py.
Can you run a training session from scratch? (Not resuming)

Thank you!
Yes, I already train from scratch.

[AlexanderHustinx]

Have you made any changes to the original resnet.py, faster_rcnn.py, or trainval_net.py?

[zychen2016]

Have you made any changes to the original resnet.py, faster_rcnn.py, or trainval_net.py?

No, Just download pascal_voc and res101_caffe and faster_1_6_10021.pth.

Using pytorch-1.0 branch

[AlexanderHustinx]

Can you try and train any model for 1 epoch and then continue training it afterwards?
Also, what OS are you running?
And which version of PyTorch? 1.0.0, 1.0.1, or higher?