Skip to content

Commit

Permalink
Formatting and wording fixes for Testing.md (#656)
Browse files Browse the repository at this point in the history
Cherry-picked from commit 00d1a11
  • Loading branch information
Mandeep Singh Grang committed Sep 27, 2019
1 parent 033fc03 commit 9ed6d1a
Showing 1 changed file with 50 additions and 34 deletions.
84 changes: 50 additions & 34 deletions clang/docs/checkedc/Testing.md
Original file line number Diff line number Diff line change
@@ -1,32 +1,36 @@
# Testing the Checked C version of LLVM/clang

LLVM/clang have two kinds of tests: developer regression tests and extended
tests. Developer regression tests are meant to be run by developers before any
check-in and are quick to run. Extended tests are run as part of continuous
tests. Developer regression tests are meant to be run by developers before any
check-in and are quick to run. Extended tests are run as part of continuous
integration testing or for changes that require extensive testing.

## Developer regression tests

We have created a new set of developer regression tests for the Checked C extension.
We have added a new target to the build system for running the test suite: check-checkedc.
The Checked C version of clang/LLVM should pass the existing clang and LLVM
regression tests and the Checked C-specific regression tests. There should
be no unexpected test failures. A developer should confirm this before committing a change.
We have created a new set of developer regression tests for the Checked C
extension. We have added a new target to the build system for running the test
suite: check-checkedc. The Checked C version of clang/LLVM should pass the
existing clang and LLVM regression tests and the Checked C-specific regression
tests. There should be no unexpected test failures. A developer should confirm
this before committing a change.

When testing a change, the testing should be sufficient for the type of change. For changes
to parsing and typechecking, it is usually sufficient to pass the Checked C and clang tests.
When testing a change, the testing should be sufficient for the type of change.
For changes to parsing and typechecking, it is usually sufficient to pass the
Checked C and clang tests.

## Running developer regressions tests

### From Visual Studio
Load the solution and the open it using the Solution explorer (View->Solution Explorer). To run tests, you can right click and build the following targets:
Load the solution and the open it using the Solution explorer (View->Solution
Explorer). To run tests, you can right click and build the following targets:

- Checked C tests: go to _CheckedC tests->check-checkedc_
- clang tests: go to _Clang tests->check-clang_
- All LLVM and clang tests: select the check-all solution (at the top level)

### From a command shell using msbuild
Set up the build system and then change to your new object directory. Use the following commands to run tests:
Set up the build system and then change to your new object directory. Use the
following commands to run tests:

- Checked C tests: `msbuild projects\checkedc-wrapper\check-checkedc.vcxproj /p:CL_MPCount=3 /m`
- Clang tests: `msbuild tools\clang\test\check-clang.vcxproj /p:CL_MPCount=3 /m`
Expand All @@ -43,24 +47,34 @@ In your build directory,
where `nnn` is replaced by the number of CPU cores that your computer has.

Note: If you use CMake with ninja, then you can simply replace `make -j nnn` in
the above commands with `ninja`. For exammple:
the above commands with `ninja`. For example:

`ninja check-checkedc`

### From a command shell using the testing harness
You can use the testing harness to run individual tests or sets of tests.
These directions are adapted from http://llvm.org/docs/TestingGuide.html#id11.
The clang-specific documentation on running tests appears to be out-of-date, so don't try to follow those directions.
The clang-specific documentation on running tests appears to be out-of-date, so
don't try to follow those directions.

- The build creates a llvm-lit.py script that you can use to run regression tests by hand.
It is placed in the same directory as your newly built clang executable. Those are placed in {your build directory}\debug\bin.
- The build creates a llvm-lit.py script that you can use to run regression
tests by hand. It is placed in the same directory as your newly built clang
executable.

- You can point the llvm-lit script at a directory to run all tests or an individual test. For example:
On Windows, the clang executable resides in:
`<BUILD_DIR>\llvm\Debug\bin` for Debug builds
`<BUILD_DIR>\llvm\Release\bin` for Release builds

llvm-lit d:\autobahn1\llvm\tools\clang\test
On Linux, the clang executable resides in:
`<BUILD_DIR>\llvm\bin`

- You can point the llvm-lit script at a directory to run all tests or an
individual test. For example:

`llvm-lit <BUILD_DIR>\llvm\tools\clang\test`

### Running lit tests for ARM
By default, the lit tests are built and run for the native X86 target. But you
By default, the lit tests are built and run for the native X86 target. But you
can also build and run them for other targets. Currently, we support lit
testing for ARM. Here is how you enable ARM lit testing:

Expand All @@ -82,25 +96,25 @@ installed on the host machine.
Once you have built the Checked C compiler, this is how you would invoke lit
for ARM testing:

`llvm-lit d:\autobahn1\llvm\projects\checkedc-wrapper\checkedc\tests -DCHECKEDC_TARGET=ARM`
`llvm-lit <SRC_DIR>\llvm\projects\checkedc-wrapper\checkedc\tests -DCHECKEDC_TARGET=ARM`

NOTE: LIT testing for ARM is only supported for tests under the directory
`llvm\projects\checkedc-wrapper\checkedc\tests`.
`<SRC_DIR>\llvm\projects\checkedc-wrapper\checkedc\tests`.

### Unexpected test failures and line endings

If you see unexpected test failures when working on Windows using unchanged
versions of the Checked C LLVM/clang repos, you should first check that you
have set your line end handling for Git properly, particularly if you see a
lot of failures. LLVM and clang have some tests that depend on line endings.
lot of failures. LLVM and clang have some tests that depend on line endings.
Those tests assume that all lines end with line feeds (the Unix line ending
convention).

The LLVM/clang project uses Subversion for source code control, which does not
alter line endings. We are using Git for source code control, which may alter
line endings on Windows, depending on how you have configured Git. Git may
alter line endings. We are using Git for source code control, which may alter
line endings on Windows, depending on how you have configured Git. Git may
alter text line endings to be carriage return/line feed (the Windows line
ending convention). It is important to ensure that Git does not do this for
ending convention). It is important to ensure that Git does not do this for
your LLVM/clang repositories.

The configuration setting `core.autocrlf` should to be set to `false`. If you
Expand All @@ -109,11 +123,12 @@ it will be set to `false`.

## Extended testing

The extended testing for LLVM/clang is somewhat complicated to set up. It
is oriented toward automated testing using servers and has the set-up complexity
that comes with that. For the Checked C implementation, there are two kinds
of extended testing that we do: testing that benchmarks that have been converted
to Checked C work and testing that the Checked C implementation has not broken existing functionality.
The extended testing for LLVM/clang is somewhat complicated to set up. It is
oriented toward automated testing using servers and has the set-up complexity
that comes with that. For the Checked C implementation, there are two kinds of
extended testing that we do: testing that benchmarks that have been converted
to Checked C work and testing that the Checked C implementation has not broken
existing functionality.

To run benchmarks, clone the
[Checked C LLVM test suite repo](https://github.com/microsoft/checkedc-llvm-test-suite)
Expand All @@ -125,16 +140,17 @@ check out the `original` branch of the
and then follow the directions in the
[README.md file](https://github.com/Microsoft/checkedc-llvm-test-suite/blob/master/README.md).
Note that the `original` branch contains tests as well as benchmarks.
It is much larger than the `master` branch. It will use more than 2 GBytes of disk space
It is much larger than the `master` branch. It will use more than 2 GBytes of disk space
and the testing will take much longer to run.


## Gotchas

Windows x86 will cause issues if executables have "install", "setup", or "update" in
their filename. It will assume the executable is an installer, and require that the user has
elevated privileges to run the program. Usually inside visual studio this manifests itself
as "WinError 740". There is more documentation on the heuristics used here:
Windows x86 will cause issues if executables have "install", "setup", or
"update" in their filename. It will assume the executable is an installer, and
require that the user has elevated privileges to run the program. Usually
inside visual studio this manifests itself as "WinError 740". There is more
documentation on the heuristics used here:
https://msdn.microsoft.com/en-us/enus/library/aa905330.aspx

We originally diagnosed this issue thanks to this stackoverflow post:
Expand Down

0 comments on commit 9ed6d1a

Please sign in to comment.