Word features in Translation

[vikrant97]

I am trying to use word level features based on the paper "Linguistic Input Features Improve Neural Machine Translation", but couldn't find any correct documentation on how to use it.
I want to use different feature vector sizes for different features.
Can anyone help me out?

[vikrant97]

I am getting this error while training.
Traning command:
python train.py -data ../data/word_with_pos_only -save_model ../models/word_with_pos_only_emb12 -layers 6 -rnn_size 512 -word_vec_size 500 -transformer_ff 2048 -heads 8 -encoder_type transformer -decoder_type transformer -position_encoding -train_steps 150000 -max_generator_batches 2 -dropout 0.3 -batch_size 4096 -batch_type tokens -normalization tokens -accum_count 2 -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 1 -max_grad_norm 0 -param_init 0 -param_init_glorot -label_smoothing 0.1 -valid_steps 5000 -save_checkpoint_steps 20000 -world_size 1 -gpu_ranks 0

example: महानगर￨NN पालिका￨NNPC अंतर्गत￨JJ दत्तात्रय￨NNPC नगर￨NNPC माध्यमिक￨NNPC स्कूल￨NN के￨PSP विद्यार्थियों￨NN ने￨PSP काल्पनिक￨JJ किला￨NN दत्तगढ़￨NNP बनाकर￨VM अपनी￨PRP कल्पनाशक्ति￨NN का￨PSP परिचय￨NN दिया￨VM

Traceback (most recent call last):
File "/home/vikrant.goyal/OpenNMT-py/train.py", line 109, in
main(opt)
File "/home/vikrant.goyal/OpenNMT-py/train.py", line 39, in main
single_main(opt, 0)
File "/home/vikrant.goyal/OpenNMT-py/onmt/train_single.py", line 127, in main
valid_steps=opt.valid_steps)
File "/home/vikrant.goyal/OpenNMT-py/onmt/trainer.py", line 249, in train
report_stats)
File "/home/vikrant.goyal/OpenNMT-py/onmt/trainer.py", line 364, in _gradient_accumulation
outputs, attns = self.model(src, tgt, src_lengths, bptt=bptt)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/vikrant.goyal/OpenNMT-py/onmt/models/model.py", line 46, in forward
memory_lengths=lengths)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/vikrant.goyal/OpenNMT-py/onmt/decoders/transformer.py", line 215, in forward
step=step)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(*input, **kwargs)
File "/home/vikrant.goyal/OpenNMT-py/onmt/decoders/transformer.py", line 69, in forward
input_norm = self.layer_norm_1(inputs)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call
result = self.forward(input, **kwargs)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/normalization.py", line 157, in forward
input, self.normalized_shape, self.weight, self.bias, self.eps)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/functional.py", line 1725, in layer_norm
torch.backends.cudnn.enabled)
RuntimeError: Given normalized_shape=[512], expected input with shape [, 512], but got input of size[227, 16, 500]

Edit: It works if i use default options of train.py but not with the above mentioned command
python train.py -data ../data/word_with_pos_only -save_model ../models/word_with_pos_only -save_checkpoint_steps 20000 -world_size 1 -gpu_ranks 0

[vikrant97]

I figured out that if I use rnn_size to be equal to the total word_vec_size (i.e including feature dimensions), then it works.
But the problem is I have to use different feature vec size for different features & the above contraint stop me doing that.
@vince62s Can you help me out in this?

[vince62s]

@vikrant97 your request is unclear to me.
each feature must have a constant vec size.
eg: if you have two features F1 and F2,then each example must have the same vec size for its F1 and same for F2. However F1 vec size can be different from F2 vec size.
are you asking something else?

[vikrant97]

@vince62s Hi, I have two problems:-

1. When using transformer model, if I choose the word_vec_size to be different than rnn_size, then it throws error. For example, use rnn_size=512 & word_vec_size=500, it will throw some error saying that it expects the word vec dimension to be equal to 512.

2. There is no option to specify different vec size for two different features say F1 & F2. The toolkit has a parameter -feat_vec_size which is a "int" and it takes only 1 value & use this as default vec size for both the features say F1 & F2 with feat_merge operation being "concat". I am saying it should take a list of feature_vec_sizes for different features F1, F2, F3....

P.S I edited the code to incorporate a list of feature_vec_sizes for different features F1, F2, F3...
but it throws some error with a conflicting issue with rnn_size.

I hope my issue is clear? If yes, can you please help me out it in these two problems?

[vince62s]

ok, then what is the issue of setting for instance emb_size 500 feat_vec_size 6, hidden 512
and if you have 2 features, it should work fine
it is not a big deal to set the feat_vec_size to the highest of what you need, is it ?

[vikrant97]

@vince62s Yes that works fine because the total vec_size will come out to be 512 only (Therefore no conflicting issues with rrn_size). But the actual problem is:
For example, if you want to use F1 (POS tag) which has a max vocab size of 50 & another feature F2 (say lemma) which has vocab size of 50k. Using the same embedding_size for both the features would not be appropriate. I would say an embedding_size of 10 would would be appropriate for F1 & F2 should have embeddings_size close to that of a word.

So I guess the trick you suggested will not work here or am I missing something?

[vince62s]

ok I see, what you are asking is in fact a duplicate of this:
#344

@bpopeters since you started to work onit, can you give him some pointers so that he can submit a PR ?
Thanks

[eduamf]

Hi Vikrant, The words’ features are back? De: Vikrant Goyal [mailto:notifications@github.com] Enviada em: quarta-feira, 21 de agosto de 2019 03:40 Para: OpenNMT/OpenNMT-py Cc: Subscribed Assunto: Re: [OpenNMT/OpenNMT-py] Word features in Translation (#1534) I am getting this error while training. Traning command: python train.py -data ../data/word_with_pos_only -save_model ../models/word_with_pos_only_emb12 -layers 6 -rnn_size 512 -word_vec_size 500 -transformer_ff 2048 -heads 8 -encoder_type transformer -decoder_type transformer -position_encoding -train_steps 150000 -max_generator_batches 2 -dropout 0.3 -batch_size 4096 -batch_type tokens -normalization tokens -accum_count 2 -optim adam -adam_beta2 0.998 -decay_method noam -warmup_steps 8000 -learning_rate 1 -max_grad_norm 0 -param_init 0 -param_init_glorot -label_smoothing 0.1 -valid_steps 5000 -save_checkpoint_steps 20000 -world_size 1 -gpu_ranks 0 example: महानगर￨NN पालिका￨NNPC अंतर्गत￨JJ दत्तात्रय￨NNPC नगर￨NNPC माध्यमिक￨NNPC स्कूल￨NN के￨PSP विद्यार्थियों￨NN ने￨PSP काल्पनिक￨JJ किला￨NN दत्तगढ़￨NNP बनाकर￨VM अपनी￨PRP कल्पनाशक्ति￨NN का￨PSP परिचय￨NN दिया￨VM Traceback (most recent call last): File "/home/vikrant.goyal/OpenNMT-py/train.py", line 109, in main(opt) File "/home/vikrant.goyal/OpenNMT-py/train.py", line 39, in main single_main(opt, 0) File "/home/vikrant.goyal/OpenNMT-py/onmt/train_single.py", line 127, in main valid_steps=opt.valid_steps) File "/home/vikrant.goyal/OpenNMT-py/onmt/trainer.py", line 249, in train report_stats) File "/home/vikrant.goyal/OpenNMT-py/onmt/trainer.py", line 364, in _gradient_accumulation outputs, attns = self.model(src, tgt, src_lengths, bptt=bptt) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/vikrant.goyal/OpenNMT-py/onmt/models/model.py", line 46, in forward memory_lengths=lengths) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/vikrant.goyal/OpenNMT-py/onmt/decoders/transformer.py", line 215, in forward step=step) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(*input, **kwargs) File "/home/vikrant.goyal/OpenNMT-py/onmt/decoders/transformer.py", line 69, in forward input_norm = self.layer_norm_1(inputs) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/module.py", line 493, in call result = self.forward(input, **kwargs) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/modules/normalization.py", line 157, in forward input, self.normalized_shape, self.weight, self.bias, self.eps) File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/nn/functional.py", line 1725, in layer_norm torch.backends.cudnn.enabled) RuntimeError: Given normalized_shape=[512], expected input with shape [, 512], but got input of size[227, 16, 500] — You are receiving this because you are subscribed to this thread. Reply to this email directly, view it on GitHub<#1534?email_source=notifications&email_token=AFTBYL2ZMTJQIOZPVBSGJFTQFTPNXA5CNFSM4INS4EN2YY3PNVWWK3TUL52HS4DFVREXG43VMVBW63LNMVXHJKTDN5WW2ZLOORPWSZGOD4YTRAQ#issuecomment-523319426>, or mute the thread<https://github.com/notifications/unsubscribe-auth/AFTBYLZ3M6Q26BR26HMTXXDQFTPNXANCNFSM4INS4ENQ>.

[vikrant97]

Hi @eduamf
The "word features" thing works if you want same feature vec size for all features.

[vikrant97]

@vince62s I have updated (locally) the OpenNMT code to incorporate word features on the source side.
It trains without any errors but fails on testing. The error seems to occur in loading the trained model.
Can you please help me out with this error?

Traceback (most recent call last):
File "translate.py", line 48, in
main(opt)
File "translate.py", line 19, in main
translator = build_translator(opt, report_score=True)
File "/home/vikrant.goyal/OpenNMT-py/onmt/translate/translator.py", line 28, in build_translator
fields, model, model_opt = load_test_model(opt)
File "/home/vikrant.goyal/OpenNMT-py/onmt/model_builder.py", line 85, in load_test_model
map_location=lambda storage, loc: storage)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/serialization.py", line 387, in load
return _load(f, map_location, pickle_module, **pickle_load_args)
File "/home/vikrant.goyal/OpenNMT-py/myenv/lib/python3.5/site-packages/torch/serialization.py", line 564, in _load
magic_number = pickle_module.load(f, **pickle_load_args)
EOFError: Ran out of input

[vince62s]

If you want to discuss some of your code, please open e [WIP] PR and then we cansse what is going right and wrong. Without knowing what you have done, difficult to help.

Most likely there could be an issue with your input features at inference.

[vikrant97]

@vince62s The issue is solved & is working on my end.
Please check the PR that I have submitted.

After this I will also look into target side features if that works and also limiting the feature vocab sizes to some particular threshold (becoz that's needed for features like lemma).
Thanks!

[Henry-E]

Based on what's written here I have a model with 4 features. So I set feat vec size to be all the same. But I still get an error.

--rnn_size 512 \
--word_vec_size 384 \
--feat_vec_size 32 \

RuntimeError: Given normalized_shape=[512], expected input with shape [*, 512], but got input of size[81, 25, 384]

When I run with regular word vector size I also get the error

--rnn_size 512 \
--word_vec_size 512 \
--feat_vec_size 32 \

RuntimeError: Given normalized_shape=[512], expected input with shape [*, 512], but got input of size[39, 102, 640]