check_bom missing in do_wordlist_crack lead to Error: wordlist grew as we read it - aborting

[hypn0s]

Hello,

We noticed the error wordlist grew as we read it - aborting with wordlists containings UTF-8 BOM combined with rules with --input-encoding=utf8 option.

After investigating, it looks like a check_bom call has been removed in a7c8fa6#diff-0fa105c10da4dc49334b36e74320a8de9996cbc2f339660ef3968bc9d9d990b3 (line 709)

This check_bom call is important as it has side effects and memmove the line pointer.

By reverting only this line, the code works as expected and no error is raised.

[magnumripper]

Interesting, thank you for reporting! I never saw it because I despise UTF-8 BOMs and also never use Windows, which is the only OS b0rken enough to produce them. I will investigate this and fix it!