Can hcxtools remove duplicate hashes from multiple captures?

[vertexgamer]

Is it possible to compare 2 or more captures and only keep unique hashes?
For example: old_capture_1 and new_capture_2 have an x amount of common hashes, which we already ran hashcat on and don't want to run it again because we are either not interested or the psw is too complex to crack.
Is it possible to save only the newly discovered hashes from new_capture_2 ?

[ZerBea]

Only hcxpcapngtool is working on dump files (pcapng/pcap/cap). All other tools are working on hash files only.
You have to convert all hashes to a hc22000 file.
Once this has been done you can use either bash tools (diff, comm, grep) or hcxhashtool to filter them.
I recommend hcxhashtool to filter by ESSID (get benefit of reuse of PBKDF2) or by MAC_AP.
While a PMKID is more or less static (same PMKID on MAC_AP & MAC_CLIENT), an EAPOL MESSAGE PAIR is completely random (depend on MAC_AP, MAC_CLIENT, randomized ANONCE, randomized SNONCE).

[vertexgamer]

Ok, thanks for the info <3
Should i filter essids based on the essid.txt extracted from the old_capture_1.pcap with hcxpcapngtool or old_hash_1.hc22000 extracted with hcxhashtool?
I noticed that the essid list generated by hcxpcapngtool is bigger; does hcxhashtool extract only the essids of captured hashes from old_hash_1.hc22000 and hcxpcapngtool extract every access point seen during the capture?

[ZerBea]

The procedure always is:
hcxpcapngtool -> hcxhashtool -> hashcat with --remove option

From now on it is easy to do a diff beetween new converted hashes and by hashcat removed ones.

[vertexgamer]

sorry m8 i'm dumb i don't get exactly what you meant.
I will just output an essid list from all previous hashes, merge it together, remove duplicates and use it as a filter for hcxhashtool and new_hash_2.hc22000
Thanks for the info on hashcat --remove, for the moment i don't need it as i'm bruteforncing only with one machine, but it's good to know in case i need to share the hashes

[vertexgamer]

oh wait fuck, --essid-list keep the essid in common, it doesn't remove them

[vertexgamer]

Ok, for future me or anyone else:

1. Extract all essid from previous hash.hc22000 with hcxhashtool
2. Combine them together and remove duplicates in your text editor of choice
3. Extract the essid from the new hash.hc22000
4. Compare the old combined essid list with the new essid list
5. Write only the unique entries from the new essid list to a new file essid.txt
6. Use essid.txt as a filter in hcxhashtool and filter new hash.hc22000 into deduplicated_hash.hc22000
7. Run hashcat with deduplicated_hash.hc22000

[ZerBea]

Exactly that is the workflow.

[vertexgamer]

btw what tool do you use to automatically keep unique lines?
I used notepad++ with the "compare" plugin, but i had to select, copy and paste the lines manually

[ZerBea]

there are several bash commands or tools beside hcxhashtool to do this:

$ cat hash.hc22000 | sort | uniq > hash_uniq.hc22000
$ comm -12 hashfile1.hc22000 hashfile2.hc22000 > comm.hc22000
$ diff hashfile1.hc22000 hashfile2.hc22000 | grep '^>' | sed -e 's/> //' > diff.hc22000

My preferred editor is Geany (GUI) or nano (command line).

[daveclifton77]

Here ya go:

cat *.22000 | sort -u > combined.22000
awk -F"*" '!_[$6]++' combined.22000 > deduped_by_ssid.22000

[ZerBea]

Another good example of how bash tools can do it much better than hcxtools could ever do.