Performing Regression Testing

When you add, remove or alter a piece of functionality, you are required to perform regression testing on your changes.


 * 1) Install DejaGnu (https://www.gnu.org/software/dejagnu/), Tcl (link in DejaGnu homepage - probably best to get version 8.6.5) and Expect (link in DejaGnu homepage). Doing this can be a pain in the ass because you have to compile all of them yourself. You'd think we were still living in 1996. However, some package managers can download DejaGnu and its dependencies for you. If that doesn't work for you, see the sections below for how to install it manually.
 * 2) Configure and build a clean version of GCC (i.e. without your changes).
 * 3) Go into that build directory and run make -k check -j [number_of_cpu_cores] . This will take a while (even longer than compiling GCC did). If you're testing a front-end, to begin with, you can do make check-c++ etcetera to check particular tests more quickly (see links below for the other commands).
 * 4) Configure GCC like you normally would when about to compile.
 * 5) Build GCC.
 * 6) Go to [build directory]/gcc/
 * 7) Run make -k check -j [number_of_cpu_cores].

Various *.sum and *.log files will be generated in the testsuite subdirectories (e.g. the result of the gcc tests will be called gcc.sum). The *.log files contain a detailed log of the compiler invocations and the corresponding results, the *.sum files summarize the results. These summaries contain status codes for all tests:


 * PASS: the test passed as expected
 * XPASS: the test unexpectedly passed
 * FAIL: the test unexpectedly failed
 * XFAIL: the test failed as expected
 * UNSUPPORTED: the test is not supported on this platform
 * ERROR: the testsuite detected an error
 * WARNING: the testsuite detected a possible problem

You can compare two .sum files by using contrib/dg-cmp-results.sh -v "" file1.sum file2.sum.

When comparing two results files you can also sometimes see NA->PASS or PASS->NA. This simply means that the line in the testsuite which tests for errors or warnings went from non-existent to existing, or existing to non-existing (i.e. you added or removed it). You can ignore these (unless you accidentally deleted them) and they are not counted as failures.

See https://gcc.gnu.org/install/test.html, https://gcc.gnu.org/wiki/Testing_GCC and https://gcc.gnu.org/onlinedocs/gccint/Testsuites.html for more in-depth information.

Common pitfalls:


 * You really should be compiling GCC on a multi-core processor. For example, on a single-core 4GHz processor, you can expect for the entire testsuite to take 6+ hours to run. Using a multi-core processor, this is a lot quicker!
 * WTF MY PATCH CAUSED 2000 FAILURES!? You probably messed something up in the build environment somewhere (that or your patch really did break stuff). Redownload the source code and dependency libraries, add your changes, and then compare again.

Installing Tcl

 * 1) Download the Tcl source from the link in the DejaGnu website.
 * 2) Extract it.
 * 3) cd into the tcl8.5.0 directory.
 * 4) run unix/configure --enable-threads.
 * 5) run make.
 * 6) run make test (probably not necessary).
 * 7) run sudo make install.

Installing Expect

 * 1) Download the Expect source from the link in the DejaGnu website.
 * 2) Extract it.
 * 3) Open a terminal in the Expect directory.
 * 4) run ./configure.
 * 5) run make.
 * 6) run sudo make install.

Installing DejaGnu

 * 1) Follow literally the same steps as you did for Expect.