Skip to content
New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

assert next_choices is not None ERROR , previously by SIGKILL error which was apparently fixed #41

Open
papsipublic opened this issue Dec 3, 2020 · 8 comments

Comments

@papsipublic
Copy link

After training with the command:

python run.py train experiments/wikisql-glove-run.jsonnet

and getting through 3990 epochs:

[2020-11-19T18:51:02] Step 39990: loss=0.8703

I tried next step:

python run eval experiments/wikisql-glove-run.jsonnet

but I go the following error:

Loading model from logdir/glove_run/model_checkpoint-00030100
0%| | 0/8421 [00:00<?, ?it/s]
Traceback (most recent call last):
File "run.py", line 109, in
main()
File "run.py", line 91, in main
infer.main(infer_config)
File "/app/ratsql/commands/infer.py", line 163, in main
inferer.infer(model, output_path, args)
File "/app/ratsql/commands/infer.py", line 71, in infer
output, args.use_heuristic)
File "/app/ratsql/commands/infer.py", line 86, in _inner_infer
decoded = self._infer_one(model, orig_item, preproc_item, beam_size, output_history, use_heuristic)
File "/app/ratsql/commands/infer.py", line 98, in _infer_one
model, data_item, preproc_item, beam_size=beam_size, max_steps=1000, from_cond=False)
File "/app/ratsql/models/spider/spider_beam_search.py", line 59, in beam_search_with_heuristics
assert next_choices is not None
AssertionError

In the logdir/glove_run I have

drwxr-xr-x. 2 root root 54 Nov 20 10:34 ie_dirs
lrwxrwxrwx. 1 root root 25 Nov 19 18:51 model_checkpoint -> model_checkpoint-00040000
-rw-r--r--. 1 root root 142281149 Nov 19 18:51 model_checkpoint-00040000
-rw-r--r--. 1 root root 240073 Nov 19 18:51 log.txt
-rw-r--r--. 1 root root 142281149 Nov 19 18:10 model_checkpoint-00039100
-rw-r--r--. 1 root root 142281149 Nov 19 17:24 model_checkpoint-00038100
-rw-r--r--. 1 root root 142281149 Nov 19 16:38 model_checkpoint-00037100

and so on.

Please advice what went wrong?

Previously, the training thrown an error related to the the fact that SIGKILL was not recognized. I replaced, following the internet fix, with a SIGTERM and an conditional to see if the object has the method.

Could

@PedroEstevesPT
Copy link
Contributor

I am experiencing the same error

@berlino
Copy link
Collaborator

berlino commented Jan 21, 2021

Can you try set 'eval_use_heuristic' to False

eval_use_heuristic: true,

@haiahaiah
Copy link

Can you try set 'eval_use_heuristic' to False

eval_use_heuristic: true,

I met the same problem as papsipublic did. And I follow your solution to set 'eval_use_heuristic' to False. But the result of WikiSQL is just 17%. I wanna ask where might be the problem. Thanks a lot~

@huybery
Copy link

huybery commented Jan 27, 2021

Can you try set 'eval_use_heuristic' to False

eval_use_heuristic: true,

I met the same problem as papsipublic did. And I follow your solution to set 'eval_use_heuristic' to False. But the result of WikiSQL is just 17%. I wanna ask where might be the problem. Thanks a lot~

I met the same problem

@drawar
Copy link

drawar commented Jul 14, 2021

I met the same problem. Any solutions?

@SkullFang
Copy link

i met too

@superquant22
Copy link

Same issue.

@mmxstar
Copy link

mmxstar commented Oct 12, 2023

same issue. Any solutions ?

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment
Labels
None yet
Projects
None yet
Development

No branches or pull requests

9 participants