Checksum MD5

[mattd9090]

Hi,
I've came across a few files that have been crawled with checksum MD5 enabled, how ever when I run the files through Powershell with get-filehash -Algorithm MD5 they don't match. I have checked other items in the index and they do match, so it only appears to be some.
Any idea what maybe causing this?
Thank you. Matt

[dadoonet]

Hey Matt

Any idea what maybe causing this?

I read somewhere that the MD5 checksum is sadly not "safe to use":

A major concern with MD5 is the potential it has for message collisions when message hash codes are inadvertently duplicated.

But this should not be the case with what you described...

Do you have an example of a file you could share here which does not generate the same MD5 Checksum? With the checksum computed by FSCrawler and the one computed within Powershell?

A workaround could be to use another algorithm...

[mattd9090]

Hi,

Thank you for your reply.

I really don't have an ideas why there would be a miss-match.

The only thing that comes to mind is that this particular index was re-indexed across from another elastic server (v7.15 to v8.3), but I do not know how that would cause this.

Unfortunately I cannot share anything from our system, I can say however one of the files that I recorded as a mismatch was a GIF.

What I can do is, I'll find the file (and any others I come across) that has the mismatch, and have FSCrawler re-crawl the folder, compare the hashes to powershell and reply with what I find.

Thank you for the suggestion, as the task Im doing is searching a list of hashes, Ill ask the team with we can get that list again but using another algorithm.

[mattd9090]

Hi,
Just an update.

A colleague of mine has said they're getting mismatch when OCR is disabled, but when enabled they match..

The mismatches I have had are apart of indexes that have had OCR disabled. OCR has always been enabled in the settings, but FSCrawler says it's disabled as it begins its crawl (only for some crawls).

I pointed my _settings at Tesseract-ocr and OCR now looks to be running as intended however I'm still getting a mismatch unlike my colleague. I'll keep playing and post any updates.

[mattd9090]

So I haven't been able to figure this one out.

Using this gif,
https://images.app.goo.gl/Wxe6MRksdZufqpWn6

Powershell get-filehash returns 43B14C45B4F719D7DB4B61E69772943D
while fscrawler file.checksum returns f85075ed8546421f7ea77e38d35bbeef

I get the same results using FSCrawler 2.7 & 2.10 and both crawlers are pointed to different ES servers.

[Jasmeet2011]

I think that powershell is giving you the hash of the file content whereas fscrawler gives hash of the file Name.
Pl verify

[mattd9090]

Quite possible.
Strange though that where I get mismatches, other systems using FScrawler are not getting these mismatches while using these same files.
I'm going to go back and play with the OCR settings to see if that changes anything.

[helsonxiao]

... fscrawler gives hash of the file Name ...

Yes, and this is the MD5 sign function used by FSCrawler.
https://github.com/dadoonet/fscrawler/blob/fe0871c4632495ac4f1edd9c25c3d819d5b373df/framework/src/main/java/fr/pilato/elasticsearch/crawler/fs/framework/SignTool.java

I'm not sure about the powershell. Maybe it's caused by the file path difference between Linux and Windows?

[dadoonet]

I'm not sure about the powershell. Maybe it's caused by the file path difference between Linux and Windows?

That could be indeed. Good idea :)

@mattd9090 could you run it in --trace mode so we might see more details?