Create fvde2john.py

[holly-o]

Added fvde2john.py for extracting FileVault hashes. Easier and more reliable hash extraction. Also will give user info such as username/password hint

[solardiz]

Thank you @holly-o!

Please also add a doc/NEWS entry. Maybe some other documentation additions are also relevant?

For the username and password hints, perhaps the script should produce an /etc/passwd-format line, putting the username into the first field, hash into the second, and password hints into the GECOS field (but first replace any potential embedded : characters in the username/hints). Then our "single crack" mode will pick those up and base candidate passwords on them.

I see there's a little bit of try/except, but perhaps it's not exhaustive for potential file read errors - in particular, the seeks look unchecked, so if a seek offset is ever beyond file size that will go unnoticed? Perhaps room for improved robustness there. But I did not review this carefully, only skimmed.

@exploide Any review from you as well?

[solardiz]

For the username and password hints, perhaps the script should produce an /etc/passwd-format line

Are these "hashes" also supported by hashcat, though? If so, I guess hashcat would insist on a bare hash? And we could need to support both output formats to be friendly to hashcat.

[solardiz]

Maybe some other documentation additions are also relevant?

Yes, this should also be added to doc/README.FileVault2, which currently only refers to @kholia's external project https://github.com/kholia/fvde2john, which is an archived repo so I assume is not currently maintained. We should probably keep that reference as a secondary option and make this new script the primary.

[exploide]

@exploide Any review from you as well?

Will do a review tomorrow.

[magnumripper]

For the username and password hints, perhaps the script should produce an /etc/passwd-format line

Are these "hashes" also supported by hashcat, though? If so, I guess hashcat would insist on a bare hash? And we could need to support both output formats to be friendly to hashcat.

FWIW hashcat does have a --username option that makes it understand a login field. Unfortunately it still chokes on fields after the hash though (at least last time I tried).

[exploide]

Left a few review comments and suggestions. :)

Looks fine overall. I just generally recommend running a code linter and formatter but no big deal here.

I haven't put much thought into the parsing logic since I'm not familiar with the FileVault data structures. And of course I did not test it in the absence of test data.

[exploide]

Are these "hashes" also supported by hashcat, though? If so, I guess hashcat would insist on a bare hash?

Yes it seems so.

$ hashcat -m 16700 --hash-info 
hashcat (v6.2.6) starting in hash-info mode

Hash Info:
==========

Hash mode #16700
  Name................: FileVault 2
  Category............: Full-Disk Encryption (FDE)
  Slow.Hash...........: Yes
  Password.Len.Min....: 0
  Password.Len.Max....: 256
  Salt.Type...........: Embedded
  Salt.Len.Min........: 0
  Salt.Len.Max........: 256
  Kernel.Type(s)......: pure
  Example.Hash.Format.: plain
  Example.Hash........: $fvde$1$16$84286044060108438487434858307513$20000$f1620ab93192112f0a23eea89b5d4df065661f974b704191
  Example.Pass........: hashcat
  Benchmark.Mask......: ?b?b?b?b?b?b?b
  Autodetect.Enabled..: Yes
  Self.Test.Enabled...: Yes
  Potfile.Enabled.....: Yes
  Custom.Plugin.......: No
  Plaintext.Encoding..: ASCII, HEX

[magnumripper]

I'm a python n00b but to me this looks like it's literally printing something like

[+] Successfully extracted hash: $fvde$1$(...)

If so, maybe the "[+] Successfully extracted hash" should go to stderr, if kept at all?

[exploide]

Yes, all messages in the script currently go to stdout. This is in contrast to most other scripts, where only the hash is printed to stdout and everything else to stderr.
I would have recommended to adapt to this behavior. Though the difference to other scripts is that this prints additional information like username and password hint.
But yes, at the moment it is untypically verbose.

[solardiz]

I think one of the script's output lines (on stdout) should be an /etc/passwd-format line without any sort of [+] Successfully extracted hash: prefix or such. So that if output is redirected to a file, that file is directly and optimally usable with john, without any need to manually select a portion of output and copy-paste.

[solardiz]

FWIW hashcat does have a --username option that makes it understand a login field. Unfortunately it still chokes on fields after the hash though (at least last time I tried).

Well, that should probably be fixed in hashcat. Maybe we should check if it still chokes on such input and send a PR fixing that.

[holly-o]

For the username and password hints, perhaps the script should produce an /etc/passwd-format line, putting the username into the first field, hash into the second, and password hints into the GECOS field (but first replace any potential embedded : characters in the username/hints). Then our "single crack" mode will pick those up and base candidate passwords on them.

For the cases where there is any potential embedded : characters or any spaces in the username/hints, is it best to remove these or replace with another character e.g. underscore, dash..?

[solardiz]

For the cases where there is any potential embedded : characters or any spaces in the username/hints, is it best to remove these or replace with another character e.g. underscore, dash..?

Spaces should be no problem, so keep those. And I suggest replace : with spaces.

Also any embedded newline characters (LF or CR) would be problematic. I suggest replace them with spaces too. I think our other *2john programs aren't careful about that, but it may be time to start to improve.

[magnumripper]

I think our other *2john programs aren't careful about that, but it may be time to start to improve.

It varies wildly. Some are careful, most are not. I think they've been fixed on a case by case basis.

[solardiz]

Spaces should be no problem, so keep those. And I suggest replace : with spaces.

I've just checked, we already have precedent - commit 4ef1ef2 implements just that kind of replacement in some *2john tools. In some others, where pathnames were used, it simply takes the basename portion (omits the directory path, which is what on Windows would have the colon after drive letter), so may still have the problem if the filename contains a colon.

As to CR/LF, I only found one example where we take care of them:

1b7ac0fdf2 run/odf2john.py         (ShaneQful    2013-07-02 20:35:04 +0100 145)         gecos = gecos.replace("\n","").replace("\r","").replace(":","")

This is now libreoffice2john.py. This script also takes os.path.basename(filename) without care for possible "bad" characters in there, but for the GECOS field it's careful like the above (but it removes the characters rather than replacing with a space - we could want to change that).

libreoffice2john.py is also usable as example for how to format the line to include hints in the GECOS field. It uses:

        sys.stdout.write("%s:$odf$*%s*%s*%s*%s*%s*%d*%s*%d*%s*%d*%s:::%s::%s\n" % \
                (os.path.basename(filename), algorithm_type, checksum_type,
                iteration_count, key_size, checksum, len(iv) / 2, iv,
                len(salt) / 2, salt, 0, binascii.hexlify(content).decode("ascii"),
                gecos, filename))

Oh, I now see it also puts the full filename into the shell field. I think we only parse that field for the shells filter option:

--shells=[-]SHELL[,..]     Load users with[out] this (these) shell(s) only

I doubt anyone would know to use that. So we could want to instead include the full filename into the GECOS field (perhaps after an added space separator) prior to the bad characters replacements.

I didn't mean to review that other script in here - it's just an example that is in some ways right and in others not.

[exploide]

Looks good :) Thanks for incorporating the changes!