optimize gix-config fuzzer performance

[Byron]

The fuzzer came up with a configuration file that has 50k sections.

Doing all these accesses to the configuration in Debug mode was slow enough for the
program to be considered stalled.

Now the fuzz-test focusses on much less of the API and limits itself to not necessarily
processing everything, which reduces the release runtime from 11 minutes to 2 seconds.

[Byron]

@silvergasp I was forced to 'neuter' the file fuzz target as it was running too slowly even in release mode, taking 11 minutes to complete on my machine. Most of the time was spent in the hasher of the hashmap, dealing with a lot of value access that I thought don't test anything that the unit tests don't cover (after all, it boils down to a hash map access).

Most of the mutations I could keep as they are generally fast enough, even though I reduced their scope a little without affecting the API.

Thinking about it, I guess the fuzzer was after increasing the runtime to the point where its not feasible anymore, even though that didn't really expose a bug in the tested API, but a shortcoming in the fuzz-program itself. To the fuzzer it doesn't matter and it can't differentiate.

I find fuzzing efficiently still quite hard - maybe the next generation of fuzzers is able to mutate the fuzz-program itself (which is basically what I did manually right now 😁).

[nathaniel-brough]

Hmm that's strange, it's not uncommon for a fuzzing test-case to timeout, but 11mins seems like an odd amount of time. When you say "11min to complete" what exactly do you mean by complete (as in run for a single test case)? Generally a fuzzer will run indefinetely or;

• until it crases
• you manually stop the fuzzer e.g. CTRL-c
• you manually set a timeout with max_total_time (distinct from the timeout option) e.g. 10min/600s cargo +nightly fuzz run fuzz_file -- -max_total_time=600

The reason that this is strange is because the default timeout for a single test case is 20min (1200s). You can change this to a smaller timeout e.g. 10s cargo +nightly fuzz run fuzz_file -- -timeout=10.

I find fuzzing efficiently still quite hard - maybe the next generation of fuzzers is able to mutate the fuzz-program itself (which is basically what I did manually right now 😁).

Yeah it's definitely a bit of an art. A super simple approach to dealing with timeouts in parsers is just to truncate the input. e.g. let input = input[..usize::min(MAX_INPUT_SIZE, input.len)];. Often this is enough, as the fuzzer will learn that producing inputs larger than MAX_INPUT_SIZE won't affect coverage at all and the inputs won't really grow beyond that. But you can also be super explicit and tell the fuzzer not to add a given input to the corpus e.g.

fuzz_target!(|input: &[u8]| {
  if input.len() > MAX_INPUT_SIZE {
    return libfuzzer_sys::Corpus::Reject;
  }
  // The reset of your fuzzer here...

  return libfuzzer_sys::Corpus::Keep;
}

There is some work going into automatically optimising test-harnesses using LLM's it's still a while off but it's pretty interesting. https://google.github.io/oss-fuzz/research/llms/target_generation/

[Byron]

Thanks for all your help and support!
I don't know if you can access it, but this here is the culprit: https://oss-fuzz.com/testcase?key=5852731717779456 .

Using the libfuzzer_sys::Corpus enum is great, and should the fuzzer still keep producing inputs that time out this test, I will most definitely use this instead.

Generally, I am actually happy about the fuzzer doing that as more often than not timeouts mean inefficient code or algorithms that ought to be fixed.

There is some work going into automatically optimising test-harnesses using LLM's it's still a while off but it's pretty interesting. https://google.github.io/oss-fuzz/research/llms/target_generation/

Definitely! I wish for starters one day I can run these things on my own computer in decent speed :D.

[nathaniel-brough]

Thanks for all your help and support!
I don't know if you can access it, but this here is the culprit: https://oss-fuzz.com/testcase?key=5852731717779456 .

Now worries. Yeah I can access all the crashes, stats and bug tracker. I think there are a few things that only the primary author sees but I'm not entirely sure what those are. I'll keep a closer eye on those timeout test cases.

Generally, I am actually happy about the fuzzer doing that as more often than not timeouts mean inefficient code or algorithms that ought to be fixed.

Yeah it's sometimes a bit of a balancing act managing performance problems with fuzzing. It's really easy to have too many false positives, and it takes a little trial and error constraining the fuzzer such that the signal to noise ratio of bugs/false positives is tolerable. But on the flip side you can easily over constrain the fuzzer and miss bugs.