Deduplicate OnDisk Corpus

[BAGUVIX456]

addresses #2434

Original PR (now closed) #2802

[tokatoka]

this is a lot ...
can you wait a bit

[tokatoka]

Hey @BAGUVIX456.
I discussed with others, but I think the previous direction is acceptable/better.
can you force push f06ed97 to the branch then we can merge after reviewing the remaining part

sorry for the confusion!

[tokatoka]

can you run ./scripts/fmt_all.sh and fix the errors in ci?

[BAGUVIX456]

yep, working on it

[tokatoka]

it seems it doesn't pass some tests

thread 'schedulers::queue::tests::test_queuecorpus' has overflowed its stack
error: test failed, to rerun pass `-p libafl --lib`

Caused by:
  process didn't exit successfully: `D:\a\LibAFL\LibAFL\target\debug\deps\libafl-6609fdc76bef418d.exe` (exit code: 0xc0000005, STATUS_ACCESS_VIOLATION)
note: test exited abnormally; to see the full output pass --nocapture to the harness.

[BAGUVIX456]

Seems like fs2 is having trouble with windows, i'll have a look

[BAGUVIX456]

Any method that modifies the name of a testcase (rename_testcase() of InMemoryOnDisk corpus or Testcase::with_filename()) runs the risk of breaking deduplication. Would just adding a warning work for now or should I do something else?
The worst case scenario is that they will end up with duplicate testcases so it is not really a critical issue, but it needs to be considered nonetheless

[BAGUVIX456]

@tokatoka the bulk of the work is completed, just need your prompt on what to do with the renaming of testcases and then we can merge this pr

[tokatoka]

Any method that modifies the name of a testcase (rename_testcase() of InMemoryOnDisk corpus or Testcase::with_filename()) runs the risk of breaking deduplication.

there's no easy way to avoid this, no? then it's fine.

and then can you run ./scripts/fmt_all.sh and fix clippy?

[domenukk]

Any method that modifies the name of a testcase (rename_testcase() of InMemoryOnDisk corpus or Testcase::with_filename()) runs the risk of breaking deduplication.

there's no easy way to avoid this, no? then it's fine.

and then can you run ./scripts/fmt_all.sh and fix clippy?

I think rename should create a new file and if the old file's counter goes to 0, remove the old file

[tokatoka]

there's still problem in windows's clippy

[BAGUVIX456]

Something wrong with the gdiplus fuzzer, will figure that out along with handling renaming

[BAGUVIX456]

@domenukk could you review whether there are any more changes that I need to do for renaming?