Should we use Csmith (+ CReduce) to test 3c for seg-faults?

[Machiry]

@mwhicks1 Do you think this is a good idea to check for some stuff that we don't handle?

[john-h-kastner]

Yeah, I've done some testing with these in the past. Makes sense to do more.

[mattmccutchen-cci]

FTR, I spent some time manually searching for ways to crash 3C for fun and found two contrived inputs that do crazy things with the preprocessor; I doubt we care about fixing them. I couldn't find a way to crash 3C without using the preprocessor (other than the already known issues in this tracker): in many cases where I forced the 3C code into an unexpected state, it just skipped a step without crashing. I guess it's good for real users if 3C doesn't crash, although it might be nice to have some warning that something unexpected happened and the results might be incorrect.

I'd be curious whether the automated tools are any good at finding inputs that crash 3C that are reasonable enough that we actually want to fix them.

My examples:

Type #include-d from different file cannot be rewritten

#include "int_star.h"
x;

where int_star.h contains:

int *

3C says Still can't rewrite declaration.This should have been made WILD during constraint generation. UNREACHABLE executed...

A PersistentSourceLoc collision causes 3C to get a FunctionVariableConstraint where it expects a PointerVariableConstraint

3c -output-postfix=checked one.h two.c -- where:
one.h:

static const int **a
#ifndef WOO
              (void)
#endif
;

two.c:

#define WOO
#include "one.h"

void two() {
  const int *b = *a;
}

3c: /home/matt/3c/clang/lib/3C/ConstraintResolver.cpp:58: CVarSet ConstraintResolver::handleDeref(CVarSet): Assertion `PVC != nullptr' failed.

[mwhicks1]

There are an infinite number of ways you can break up the syntactic structure of a program with macros so as to confuse source-code processing tools. I'd prefer to avoid worrying about cases like these until we are faced with code that must use them.

BTW, @Machiry has two students at Purdue who are starting to test 3C using CSmith and CReduce.