Detect OpenMP Races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded C program written with OpenMP.

Prerequisites

• This tutorial assumes you have successfully installed the Coderrect software following the quick start.

• Our sample code relies on OpenMP to achieve parallelism. You will need a compiler that supports OpenMP. We will be using gcc, but clang and other modern alternatives also have OpenMP support. 

Detect a race in pi.c

We will start by detecting a race in a small single file program pi.c. The program is designed to compute pi=3.1415926…. We provide this example program in the Examples directory under Coderrect installation, or you can copy the code below to your system.

//pi.c
#include <omp.h>
#include <stdio.h>

#define N 1000000000

int main () {

    double delta = 1.0/(double) N;
    int MAX_THREADS = omp_get_max_threads();
    // Compute parallel compute times for 1-MAX_THREADS
    for (int j=1; j<= MAX_THREADS; j++) {

        printf(" running on %d threads: ", j);
        omp_set_num_threads(j);

        double sum = 0.0;
        double start = omp_get_wtime();

        #pragma omp parallel for //reduction(+:sum)
        for (int i=0; i < N; i++) {
            double x = (i+0.5)*delta;
            sum += 4.0 / (1.0+x*x);
        }

        // Out of the parallel region, finalize computation
        double pi = delta * sum;
        double time_lapse = omp_get_wtime() - start;
        printf("PI = %.12g computed in %.4g secondsn", pi, time_lapse);
    }
}

Check that pi.c can be compiled and run with the following commands

gcc -fopenmp pi.c -o pi
./pi

You should see output that looks something like:

running on 1 threads: PI = 3.141592653589971 computed in 12.84 seconds
running on 2 threads: PI = 3.141593993623682 computed in 6.928 seconds
running on 3 threads: PI = 3.141594228301372 computed in 7.741 seconds
running on 4 threads: PI = 3.141595112697573 computed in 8.376 seconds

As you can see from the results, running on different number of threads generated different values of PI, indicating the existence of a concurrency bug.

Run Coderrect

The easiest way to run the tool is by passing the build command to coderrect:

coderrect -t gcc -fopenmp pi.c

Remember, the command to build pi.c was gcc -fopenmp pi.c.

-t switch is used to generate a quick summary report in terminal.

Calling coderrect in this way ensures all the required compilation flags can be passed on the command line. For a project using a build system such as make, coderrect tool can be called with the same build command used to build the project. For an example: checkout out detecting races in a Makefile-based project.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in a browser.

Terminal Report

The terminal report for pi.c should look something like the following:

==== Found a race between: 
line 22, column 13 in test.c AND line 22, column 17 in test.c
Shared variable:
 at line 16 of test.c
 16|        double sum = 0.0;
Thread 1:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }
 24|
>>>Stacktrace:
Thread 2:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }
 24|
>>>Stacktrace:

                 1 OpenMP races

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 22, column 13 in test.c AND line 22, column 17 in test.c

Next the report shows the variable name and location on which the race occurs.

Shared variable:
 at line 16 of test.c
 16|        double sum = 0.0;

This shows that the race occurs on the variable sum declared on line 16.

Going and finding the race location in the code may be a little tedious so the report also shows a preview of the file at that location.

Thread 1:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }

The code snippet shows that this racing access is a write to sum as part of an OpenMP parallel for loop.

Taking a closer look at the source code we can see the root cause is the commented out “reduction”. 

#pragma omp parallel for //reduction(+:sum)

Un-commenting reduction(+:sum) removes the data race on sum and allows the program to calculate pi correctly.

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

We can also save the race report to a directory specified via the command option-o <dir>. 

coderrect -o report gcc -fopenmp pi.c

This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Detect pthread races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded C++ program written with the POSIX threads (Pthreads) library.

coderrect g++ pthread-race.cc -lpthread

Prerequisites

This tutorial assumes you have successfully installed the Coderrect software following the quick start.

Detecting races in pthread-race.cc

Our sample code pthread-race.cc starts two children threads in the main method. Each child thread increments a shared variable “x” by one, but there is a race.

Copy the source for pthread-test.cc below to your system.

#include <pthread.h>

#include <cstdlib>
#include <iostream>

using namespace std;
int x = 0;

void *PrintHello(void *threadid) {
    long tid;
    tid = (long)threadid;
    x++;
    cout << "Hello World! Thread ID, " << tid << endl;
    pthread_exit(NULL);
}

pthread_t load_data_in_thread(long id) {
    pthread_t thread;
    void *arg = (void *)id;
    int rc = pthread_create(&thread, NULL, PrintHello, arg);
    if (rc) {
        cout << "Error:unable to create thread," << rc << endl;
        exit(-1);
    }
    return thread;
}

int main() {
    pthread_t thread1, thread2;

    cout << "main() : creating thread 1 " << endl;
    thread1 = load_data_in_thread(1);
    cout << "main() : creating thread 2 " << endl;
    thread2 = load_data_in_thread(2);
    pthread_join(thread1, 0);
    // pthread_join(thread2,0);
    cout << "Final value of x: " << x << endl;
}

Check that pthread-race.cc can be compiled and run with the following commands

g++ pthread-race.cc -lpthread
./a.out

If run ./a.out multiple times, you should see output that looks something like:

main() : creating thread 1 
main() : creating thread 2 
Hello World! Thread ID, 1
Final value of x: 1
Hello World! Thread ID, 2

Run Coderrect

The easiest way to get started on a single file is to run

coderrect -t g++ pthread-race.cc -lpthread

-t switch is used to generate a quick summary report in terminal, for more information on Coderrect configurations, please check out reference page.

This will automatically detect the races and report the follow in the terminal:

==== Found a race between: 
line 12, column 5 in pthread-race.cc  AND  line 38, column 41 in pthread-race.cc
Shared variable: 
 at line 7 of pthread-race.cc
 7|int x =0;
Thread 1: 
pthread-race.cc@12:5
Code snippet: 
 10| long tid;
 11|    tid = (long)threadid;
>12|    x++;
 13|    cout << "Hello World! Thread ID, " << tid << endl;
 14|    pthread_exit(NULL);
>>>Stack Trace:
>>>PrintHello(void*) [pthread-race.cc:20]
Thread 2: 
pthread-race.cc@38:41
Code snippet: 
 36|      //pthread_join(thread2,0);
 37|
>38|        cout << "Final value of x: " << x << endl;
 39|}
>>>Stack Trace:
>>>main()
>>>  std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) [pthread-race.cc:38]
>>>    std::char_traits<char>::length(char const*) [/usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/ostream:562]
detected 1 races in total.

2020/02/25 01:18:23 Generating the race report ...
To check the race report, please open './index.html' in your browser

Interpret the Results

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 12, column 5 in pthread-race.cc  AND  line 38, column 41 in pthread-race.cc

Next the report shows the name and location of the variable on which the race occurs.

Shared variable: 
 at line 7 of pthread-race.cc
 7|int x =0;

This shows that the race occurs on the variable x declared on line 7.

Next the tool reports information about the two unsynchronized accesses to x.
For each of the two accesses a location, code snippet, and stack trace is shown.

The location shows the file, line, and column of the access.

pthread-race.cc@12:5
pthread-race.cc@38:41

So the above access occurs in pthread-race.cc at line 12 column 5 and at line 38 column 41, respectively.

Going and finding this location in the code may be a little tedious so the report also shows a preview of the file at that location.

Code snippet: 
 10| long tid;
 11|    tid = (long)threadid;
>12|    x++;
 13|    cout &lt;&lt; "Hello World! Thread ID, " << tid << endl;
 14|    pthread_exit(NULL);

The code snippet shows that this access is an unsynchronized write to x in each child thread.

The last piece of information shown for each access is the stack trace.

>>>Stack Trace:
>>>PrintHello(void*) [pthread-race.cc:20]

>>>main()
>>> std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) [pthread-race.cc:38]
>>> std::char_traits<char>::length(char const*) [/usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/ostream:562]

The stack trace shows the call stack under which the racing access occurred.
Each line in the call stack shows the name of the function, and the location the function was called from.

In the example above the stack trace shows PrintHello being called from line 20 in pthread-race.cc.

HTML Report

The full report can be viewed in a browser.

Detect races in GPU kernels

This tutorial showcases Coderrect on detecting block-level and warp-level race hazards in GPU/CUDA kernels. Both examples are from NVIDIA’s official documentation: block_error.cu and warp_error.cu. Note that if you do not already have nvidia-cuda-toolkit installed on your machine you must use the command listed below.

$ sudo apt install nvidia-cuda-toolkit
$ coderrect -t nvcc block_error.cu

==== Found a race between: 
line 9, column 5 in block_error.cu AND line 14, column 25 in block_error.cu
Shared variable: 
smem at line 3 of block_error.cu
 3|__shared__ int smem[THREADS];
Thread 1: 
 7|{
 8|    int tx = threadIdx.x;
>9|    smem[tx] = data_in[tx] + tx;
 10|
 11|    if (tx == 0) {
>>>Stack Trace:
Thread 2: 
 12|        *sum_out = 0;
 13|        for (int i = 0; i < THREADS; ++i)
>14|            *sum_out += smem[i];
 15|    }
 16|}
>>>Stack Trace:
The OpenMP region this bug occurs:
/CUDA/benchmarks/t/block_error.cu
>27|    sumKernel<<<1, THREADS>>>(data_in, sum_out);
 28|    cudaDeviceSynchronize();
 29|
 30|    cudaFree(data_in);
 31|    cudaFree(sum_out);
 32|    return 0;
Gets called from:
>>>main
detected 1 races in total.
To check the race report, please open '/CUDA/benchmarks/t/.coderrect/report/index.html' in your browser

$ coderrect -t nvcc wrap_error.cu

==== Found a race between: 
line 12, column 5 in wrap_error.cu AND line 19, column 32 in wrap_error.cu
Shared variable: 
smem_first at line 5 of wrap_error.cu
 5|__shared__ int smem_first[THREADS];
Thread 1: 
 10|{
 11|    int tx = threadIdx.x;
>12|    smem_first[tx] = data_in[tx] + tx;
 13|    //__syncwarp();
 14|    if (tx % WARP_SIZE == 0) {
>>>Stack Trace:
Thread 2: 
 17|        smem_second[wx] = 0;
 18|        for (int i = 0; i < WARP_SIZE; ++i)
>19|            smem_second[wx] += smem_first[wx * WARP_SIZE + i];
 20|    }
 21|
>>>Stack Trace:
The OpenMP region this bug occurs:
/CUDA/benchmarks/t/wrap_error.cu
>40|    sumKernel<<<1, THREADS>>>(data_in, sum_out);
 41|    cudaDeviceSynchronize();
 42|
 43|    cudaFree(data_in);
 44|    cudaFree(sum_out);
 45|    return 0;

Note that in the above code line 13 is commented out, which disables the warp-level synchronization. 

13| //__syncwarp();

If line 13 is uncommented, the race will be fixed, and the tool will report no races: 

detected 0 races in total.

Detect races in a Makefile-based project

This tutorial shows how to use Coderrect to detect races in a Makefile-based project using pbzip2-0.9.4 as the example. pbzip2 is a parallel implementation of the bzip2 file compressor, and it contains a known race condition in version 0.9.4.

git clone https://github.com/sctbenchmarks/sctbenchmarks.git
cd sctbenchmarks/1CB-0-2/pbzip2-0.9.4/bzip2-1.0.6
make clean
make
cd ..
cd pbzip2-0.9.4
make clean
coderrect make

Prerequisites

This tutorial assumes you have successfully installed the Coderrect software following the quick start.

Background

In pbzip2, the program will spawn consumer threads that (de)compress the input file and spawn an output thread that writes data to the output file. However, the main thread only joins the output thread but does not join the consumer threads. So there is an order violation bug between the time when the main thread is destroying resources and the time when consumer threads are using the resources.

An interleaving that triggers the error looks like:

void main(...) { 
  ...
  for (i=0; i < numCPU; i++) {              
    ret = pthread_create(..., consumer, 
                              fifo);  
    ...             
  }
  ret = pthread_create(..., fileWriter, 
                            OutFilename);
  ...
  // start reading in data
  producer(..., fifo);
  ...
  // wait until exit of thread
  pthread_join(output, NULL);
  ...
  fifo->empty = 1;
  ...
  // reclaim memory
  queueDelete(fifo);
  fifo = NULL;



}

void *decompress_consumer(void *q) {
  ...
  for (;;) {


















    pthread_mutex_lock(fifo->mut);
    ...
  }
}

Since queueDelete will release the fifo queue used by consumer threads, the access on fifo->mut will result in a segmentation fault.

Detect the race using Coderrect

Make sure the code can be compiled

cd sctbenchmarks/1CB-0-2/pbzip2-0.9.4/pbzip2-0.9.4
make

You should see a pbzip2 executable under the same folder.

Run Coderrect

make clean 
coderrect -t -o report make

NOTE: The make clean command is to ensure there is no pre-built binaries so that Coderrect is able to analyze every piece of source code in the project.

The coderrect -t -o report make command will compile and analyze the problem, the reported races is stored under ./report directory as specified by -o option.

-t switch is used to generate a quick summary report in terminal.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in the browser.

Terminal Report

The terminal races reports looks like following:

==== Found a race between: 
line 1048, column 3 in pbzip2.cpp  AND  line 553, column 28 in pbzip2.cpp
Shared variable: 
 at line 991 of pbzip2.cpp
 991| q = new queue;
Thread 1: 
pbzip2.cpp@1048:3
Code snippet: 
 1046| pthread_mutex_destroy(q->mut);
 1047| delete q->mut;
>1048| q->mut = NULL;
 1049| }
 1050|
>>>Stack Trace:
>>>main
>>>  queueDelete(queue*) [pbzip2.cpp:1912]
Thread 2: 
pbzip2.cpp@553:28
Code snippet: 
 551| for (;;)
 552| {
>553| pthread_mutex_lock(fifo->mut);
 554| while (fifo->empty)
 555| {
>>>Stack Trace:
>>>pthread_create [pbzip2.cpp:1818]
>>>  consumer_decompress(void*) [pbzip2.cpp:1818]

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 1048, column 3 in pbzip2.cpp  AND  line 553, column 28 in pbzip2.cpp

Next the report shows the name and location of the variable on which the race occurs.

Shared variable: 
 at line 991 of pbzip2.cpp
 991| q = new queue;

This shows that the race occurs on the variable queue allocated at line 991.

Next the tool reports information about the two unsynchronized accesses to queue.
For each of the two accesses a location, code snippet, and stack trace is shown.

The location shows the file, line, and column of the access.

Thread 1: 
pbzip2.cpp@1048:3

So the above access occurs in pbzip2.cpp at line 1048 column 3.
The report also shows a preview of the file at that location.

Code snippet: 
 1046| pthread_mutex_destroy(q->mut);
 1047| delete q->mut;
>1048| q->mut = NULL;
 1049| }
 1050|

The code snippet shows that this access is a write to q->mut (set it to NULL).

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

Detect Fortran OpenMP Races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded Fortran program written with OpenMP.

Prerequisites

• This tutorial assumes you have successfully installed the Coderrect software following the quick start.

• Our sample code relies on OpenMP to achieve parallelism. You will need a compiler that supports OpenMP. We will be using gfortran, but other modern alternatives should also have OpenMP support. 

Detecting a race

We will be using a benchmark from DataRaceBench, a suite of OpenMP data race benchmarks designed to evaluate the effectiveness of data race detection tools developed by a group at Lawrence Livermore National Lab. DataRaceBench is full of great test cases (try using it to evaluate coderrect!).

We will be using the DRB001-antidep1-orig-yes.f95 case from DataRaceBench version 1.3.0.1. You can get the source code from the DataRaceBench repository on github, but we have included a snippet here for convenience.

!!!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~!!!
!!! Copyright (c) 2017-20, Lawrence Livermore National Security, LLC
!!! and DataRaceBench project contributors. See the DataRaceBench/COPYRIGHT file for details.
!!!
!!! SPDX-License-Identifier: (BSD-3-Clause)
!!!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~!!!

!A loop with loop-carried anti-dependence.
!Data race pair: a[i+1]@25:9 vs. a[i]@25:16

program DRB001_antidep1_orig_yes
use omp_lib
    implicit none
    integer :: i, len
    integer :: a(1000)

    len = 1000

    do i = 1, len
        a(i) = i
    end do

    !$omp parallel do
    do i = 1, len-1
        a(i) = a(i+1) + 1
    end do
    !$omp end parallel do

    print 100, a(500)
    100 format ('a(500)=',i3)
end program

Start by checking that the program compiles successfully on your machine. Coderrect works by intercepting compiler calls. If the code cannot be compiled, Coderrect cannot run it’s analysis. The DRB001 benchmark can be built with the following command

gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001
./DRB001

You should see output that looks something like:

a(500)=502

Although your results may vary because there is a data race in this code. In the parallel for loop, the value of each iteration i depends on the next iteration i+1.

This means that thread 0 could be executing a[0] = a[1] + 1 at the same time thread 1 is running a[1] = a[2] + 1. Both threads are accessing a[1] in parallel, causing a data race.

Run Coderrect

The easiest way to run the tool is by passing the build command to coderrect:

coderrect -t gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001

Remember, the command to build DRB001 was gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001.

the -t switch is used to generate a quick summary report in terminal.

Calling coderrect in this way ensures all the required compilation flags can be passed on the command line. For a project using a build system such as make, coderrect tool can be called with the same build command used to build the project. For an example: checkout out detecting races in a Makefile-based project.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in a browser.

Terminal Report

The terminal report for DRB001 should look something like the following:

==== Found a data race between: 
line 25, column 0 in DRB001-antidep1-orig-yes.f95 AND line 25, column 1 in DRB001-antidep1-orig-yes.f95
Shared variable:
 at line 0 of 
 0|
Thread 1:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do
>>>Stacktrace:
Thread 2:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do
>>>Stacktrace:

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 25, column 0 in DRB001-antidep1-orig-yes.f95 AND line 25, column 1 in DRB001-antidep1-orig-yes.f95

Next the report shows the variable name and location on which the race occurs. (Though this is sometimes not present for fortran programs)

Shared variable:
 ...

Going and finding the race location in the code may be a little tedious so the report also shows a preview of the file at that location.

Thread 1:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do

The code snippet shows that this racing access is a write to a(i) as part of an OpenMP parallel for loop.

This is the race we expected to find for this DataRaceBench case. You can try running coderrect in the same way on the other DataRaceBench fortran benchmarks at dataracebench/micro-benchmarks-fortran.

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

We can also save the race report to a directory specified via the command option-o <dir>. 

coderrect -o report gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001

This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Exclude Uninteresting Code Paths

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

By default, Coderrect traverses every function that is reachable from the main function (or from any of the user-specified entry functions). However, in many cases, users might regard certain functions as uninteresting in their own projects and therefore do not want Coderrect to analyze those functions. This can be done by setting configuration for “skipFunctions” in .coderrect.json:

//.coderrect.json
"skipFunctions":[
        "print_msg"
] 

In the above example, the function “print_msg” will not be analyzed by Coderrect.

Analyzing Interesting Indirect Calls

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

To make Coderrect fast, in the default setting Coderrect only analyzes a limited number of indirect function calls (functions that are invoked through function pointers) for each call site. It is because that the number of targets that can be pointed by the function pointer can be large and analyzing each of them is time-consuming. This works well for common cases in practice. However, in cases where a call site has many targets, users can configure “indirectFunctions” to analyze all possible targets called in a function. For example:

"indirectFunctions":[
        "OpBase_Consume"
] 

In the above, OpBase_Consume is a function from RedisGraph and it contains an indirect call op->consume(op) with a large number of possible call targets:

inline Record OpBase_Consume(OpBase *op) {
        return op->consume(op);
}

The configuration instructs Coderrect to analyze all possible indirect call targets in the function “OpBase_Consume“.

Define Custom Lock APIs

Quick Start

To make Coderrect understand your custom lock APIs, please open your configuration file (the default one is $installDir/conf/coderrect.json), and add the following lines:

{
        ... ...
        lockUnlockFunctions: {
            "MyMutexLockFunc": "MyMutexUnlockFunc",
            "MySpinLockFunc": "MySpinUnlockFunc"
        }
}

Introduction

Lock APIs (e.g. pthread_mutex) are the most common synchronization mechanism used by developers. Coderrect has the built-in support for std::lock, std::mutex, pthread_mutex, and pthread_spin_lock. However, developers may use third-party libraries that have their own lock APIs. Furthermore, developers may create their own lock APIs to fit the special needs. 

Coderrect relies on the configuration “lockUnlockFunctions” to recognize these proprietary APIs.

lockUnlockFunctions

The code below creates a naive custom spin lock and protects simultaneous access to the global variable “x” by using it. 

#include <iostream>
#include <atomic>
#include <thread>

struct my_spinlock_t {
    std::atomic<bool> a;

    my_spinlock_t() : a(false) {}
};

void MySpinLock(my_spinlock_t* lock) {
    bool expected = false;
    while (!lock->a.compare_exchange_strong(expected, true))
    ;
}

void MySpinUnlock(my_spinlock_t* lock) {
    lock->a = false;
}

my_spinlock_t lock;
int x;


void thread_function() {
    MySpinLock(&lock);
    x++;
    std::cout << x << "n";
    MySpinUnlock(&lock);
}


int main ()
{
    std::thread first (thread_function);
    std::thread second (thread_function);
    first.join();
    second.join();
    return 0;
}

Let’s run Coderrect with the default configuration:

Coderrect detects a race because it doesn’t understand the meaning of MySpinLock and MySpinUnlcok and determines “x” are accessed by two threads simultaneously without synchronization.

Now, let’s tell Coderrect the meaning of MySpinLock/MySpinUnlock with a custom local configuration file lockunlockfunc.json:

{
    "lockUnlockFunctions": {
         "MySpinLock": "MySpinUnlock"
    }
}

The lockUnlockFunctions is a list of key-value pairs. Each pair has the name of the lock function as the key and the name of the unlock function as the value.

Let’s apply this configuration file to Coderrect and try again

You tell Coderrect to load and overlap the configuration file “lockunlockfunc.json” to the default coderrect.json using “-conf” option. Now, Coderrect knows MySpinLock and MySpinUnlock are a pair of lock-unlock functions and makes correct reasoning of the thread interleave.

Support Version

Coderrect 0.7.0+

Common Configurations

Detect races in an open source project Memcached

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Coderrect detected 3 new races (2 of them were confirmed) in memcached. To detect the reported bugs using coderrect, download and checkout the buggy version of memcached using the following commands.

$ git clone https://github.com/memcached/memcached.git
$ cd memcached
$ git checkout 82029ecc9b3dd0f57b3f9ab9761f44714cceed6f

Detect the race

1. Build memcached using coderrect

# install dependencies
$ apt install libevent-dev
# configure memcached
$ ./autogen.sh && ./configure
# build memcached using coderrect
$ coderrect -t make

The coderrect -t make command will compile and analyze the program automatically. 

2. Detect races using coderrect

After compilation, coderrect automatically detects and lists all the potential targets to analyzed as follows: 

1) timedrun
2) sizes
3) memcached-debug
4) memcached
5) testapp
Please select binaries by entering their ordinal numbers (e.g. 1,2,6):

Select 3) memcached-debug as the target to detect races on the debug version of memcached.

Interpret the Results

The coderrect tool generates a comprehensive report that can be viewed in a browser. 

HTML Report

To view the full report, open ‘.coderrect/report/index.html‘ in your browser.

The HTML report looks like the following picture.

Terminal Report

To get a quick overview of the detected races, coderrect can also report a summary of the most interesting races in the terminal (with -t flag, checkout all coderrect options). The terminal races report looks like the following:

==== Found a race between: 
line 162, column 5 in crawler.c AND line 1464, column 16 in items.c
Shared variable:
 at line 1577 of items.c
 1577|        calloc(1, sizeof(struct crawler_expired_data));
Thread 1:
 160|    pthread_mutex_lock(&d->lock);
 161|    d->crawlerstats[slab_cls].end_time = current_time;
>162|    d->crawlerstats[slab_cls].run_complete = true;
 163|    pthread_mutex_unlock(&d->lock);
 164|}
>>>Stacktrace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    lru_crawler_class_done [crawler.c:378]
>>>      crawler_expired_doneclass [crawler.c:350]
Thread 2:
 1462|        crawlerstats_t *s = &cdata->crawlerstats[i];
 1463|        /* We've not successfully kicked off a crawl yet. */
>1464|        if (s->run_complete) {
 1465|            char *lru_name = "na";
 1466|            pthread_mutex_lock(&cdata->lock);
>>>Stacktrace:
>>>pthread_create [items.c:1703]
>>>  lru_maintainer_thread [items.c:1703]
>>>    lru_maintainer_crawler_check [items.c:1647]

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 162, column 5 in crawler.c AND line 1464, column 16 in items.c

Next the report shows the name and location of the variable on which the race occurs.

Shared variable:
 at line 1577 of items.c
 1577|        calloc(1, sizeof(struct crawler_expired_data));

For example, the above result shows that the race occurs on the variable allocated at line 1577 in items.c file.

Next the tool reports information about the two unsynchronized accesses to the shared variable. For each of the two accesses, the code snippet, and the stack trace is shown.

Since finding the reported location in the code may be a little tedious, the report shows a preview of the file at that location.

Thread 1:
 160|    pthread_mutex_lock(&d->lock);
 161|    d->crawlerstats[slab_cls].end_time = current_time;
>162|    d->crawlerstats[slab_cls].run_complete = true;
 163|    pthread_mutex_unlock(&d->lock);
 164|}

The code snippet shows that the race is on variable crawlerstats[slab_cls].run_complete. Coderrect also shows the stack trace that triggers the race to make validation on the race simpler.

>>>Stacktrace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    lru_crawler_class_done [crawler.c:378]
>>>      crawler_expired_doneclass [crawler.c:350]

Detecting Races in Redis

Redis is a popular in-memory database that persists on disk. Redis has been heavily adopted and is widely used. The GitHub repository has over 43,000 stars, over 18,000 questions have been tagged with [redis] on StackOverflow, and there are various meetups and conferences for Redis every year.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Detect the race

Redis is easy to build. To detect races on redis, simply run the following:

git clone -b 6.0.0 https://github.com/antirez/redis.git
cd redis
coderrect -t -o report -e redis-server,redis-benchmark make -j

• First, we clone the buggy release (6.0.0). 
• Next, we cd into the redis directory and run the Coderrect tool to detect races. 
• The -t flag will print the race report to the terminal so that we can quickly investigate some of the races. 
• The -o report flag tells the tool to generate a report and place it into a directory named “report”. 
• Next, Redis builds multiple executables by default (redis-server, redis-benchmark, and redis-cli). Coderrect is capable of analyzing one or more targets at once. Here we specify that Coderrect should analyze both redis-server and red-benchmark with the flag  -e redis-server,redis-benchmark. 
• Lastly, we add the command make -j to build redis.

The analysis time for each project will be printed to the terminal as is shown below:

Analyzing /path/to/redis/src/redis-benchmark ...
 - ✔ [00m:01s] Loading IR From File                    
 - ✔ [00m:02s] Running Compiler Optimization Passes (Phase I)                                              
 - ✔ [00m:00s] Canonicalizing Loops                    
 - ✔ [00m:00s] Propagating Constants                     
 - ✔ [00m:00s] Running Compiler Optimization Passes (Phase II)                                               
 - ✔ [00m:00s] Running Pointer Analysis                        
 - ✔ [00m:09s] Building Static Happens-Before Graph                                    
 - ✔ [00m:11s] Detecting Races               
 - ✔ [00m:00s] Scanning for additional OpenMP Regions                                                                        

Analyzing /path/to/redis/src/redis-server ...
 - ✔ [00m:04s] Loading IR From File                    
...


----------------------------The summary of races in redis-server------------------------

 12 shared data races

----------------------------The summary of races in redis-benchmark------------------------

  3 shared data races

Coderrect provide an option to list all potential target binaries and specify which binaries should be analyzed with the following command.

coderrect make

Coderrect will automatically detect which binaries were built using make and list the potential targets to be analyzed as shown below.

The project creates multiple executables. Please select one from the list below
to detect races. 

In the future, you can specify the executable using the option "-e" if you know 
which ones you want to analyze. 

    coderrect -e your_executable_name1,your_executable_name2 your_build_command_line

 1) src/redis-benchmark
 2) src/redis-cli
 3) src/redis-server
 4) deps/lua/src/luac
 5) deps/lua/src/lua

Please select binaries by entering their ordinal numbers (e.g. 1,2,6):1,2,3

In addition, Coderrect allow users to analyze all generated binaries by using -analyzeAllBinaries flag to command line. Coderrect will skip the asking stage and automatically analyze all binaries.

coderrect -analyzeAllBinaries make

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing when run with -t. The tool also generates a more comprehensive report that can be viewed in the browser. 

Terminal Report

Next we will explain an example of a race reported in redis-server, as well as how to debug the race based on the provided information:

==== Found a race between: 
line 888, column 9 in debug.c AND line 885, column 16 in debug.c
Shared variable:
 at line 72 of server.c
 72|struct redisServer server; /* Server global state */
Thread 1:
 886|        serverLogRaw(LL_WARNING|LL_RAW,
 887|        "nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n");
>888|        server.bug_report_start = 1;
 889|    }
 890|}
>>>Stacktrace:
>>>pthread_create [bio.c:123]
>>>  bioProcessBackgroundJobs [bio.c:123]
>>>    lazyfreeFreeObjectFromBioThread [bio.c:209]
>>>      decrRefCount [lazyfree.c:130]
>>>        freeListObject [object.c:365]
>>>          _serverPanic [object.c:291]
>>>            bugReportStart [debug.c:873]
Thread 2:
 883|
 884|void bugReportStart(void) {
>885|    if (server.bug_report_start == 0) {
 886|        serverLogRaw(LL_WARNING|LL_RAW,
 887|        "nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n");
>>>Stacktrace:
>>>pthread_create [networking.c:2942]
>>>  IOThreadMain [networking.c:2942]
>>>    _serverAssert [networking.c:2891]
>>>      bugReportStart [debug.c:791]

At a glance we can quickly see that the race is between the lines 885 and 888 in the debug.c file. Line 888 writes to server.bug_report_start and line 885 reads server.bug_report_start. To confirm this race we need to show 1) both accesses can be made in parallel, and 2) the parallel accesses are being made to the same server.bug_report_start object.

884|void bugReportStart(void) {
885|  if (server.bug_report_start == 0) {
886|    serverLogRaw(LL_WARNING|LL_RAW,
887|    "nn=== REDIS BUG REPORT START: ...n");
888|    server.bug_report_start = 1;
889|  }
890|}

To prove that 1) both accesses can be made in parallel, we need to show that thread 1 and 2 can both call the bugReportStart function at the same time. We can check the stack trace to see if there are any synchronizations between the threads after they are spawned.

Thread 1
>>>Stacktrace:
>>>pthread_create [bio.c:123]
>>>  bioProcessBackgroundJobs [bio.c:123]
>>>    lazyfreeFreeObjectFromBioThread [bio.c:209]
>>>      decrRefCount [lazyfree.c:130]
>>>        freeListObject [object.c:365]
>>>          _serverPanic [object.c:291]
>>>            bugReportStart [debug.c:873]
Thread 2:
>>>Stacktrace:
>>>pthread_create [networking.c:2942]
>>>  IOThreadMain [networking.c:2942]
>>>    _serverAssert [networking.c:2891]
>>>      bugReportStart [debug.c:791]

The first stack trace shows that thread 1 is spawned at bio.c:123 inside of the bioInit function. The second stack trace shows that thread 2 is spawned at networking.c:2942 in the initThreadedIO function. By looking one level up the stack trace, we can see that both bioInit and initThreadedIO are called one after the other from InitServerLast in server.c. 

void InitServerLast() {
    bioInit();
    initThreadedIO();

If we check the body of the bioInit function we see there are no synchronizations or joins after thread 1 is spawned. This means thread 1 will continue to execute even after the main thread has returned from bioInit and entered initThreadIO. Then, while in initThreadIO the main thread spawns thread 2 in parallel with thread 1. Thus we have shown that both threads do indeed run in parallel.

Next we must show that 2) the parallel accesses are being made to the same server.bug_report_start object. The report also shows us the creation site of the shared object on which the race occurs (in this case server).

Shared variable:
 at line 72 of server.c

By inspecting server.c:72 we see that server is actually a global variable! This means that that both threads are indeed accessing the same object in memory.

72| struct redisServer server; /* Server global state */

So, by showing that 1) both accesses can be made in parallel and 2) the parallel accesses are being made to the same server.bug_report_startobject we have confirmed this race in redis.

HTML Report

The terminal is great to get a quick idea about what races are reported, but a full and more detailed report can be viewed in the browser.

When we initially ran coderrect we included the -o report flag. This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Depending on your system this can be done through the terminal with some command. E.g. browse report/index.html on Gnome based systems or open report/index.html on Mac.

Detecting Races in RedisGraph

RedisGraph is a popular graph database module of Redis that supports using linear algebra to query the graph. This tutorial shows how to run coderrect on RedisGraph for finding potential races step by step.

1. Download RedisGraph and setup dependency:

$ git clone --recurse-submodules https://github.com/RedisGraph/RedisGraph.git
$ cd RedisGraph
$ apt-get install build-essential cmake m4 automake peg libtool autoconf

2. Run coderrect:

$ coderrect -t make

Running coderrect -t make builds the RedisGraph module redisgraph.so, which is a dynamic library. To analyze the library, Coderrect prompts a list of APIs for users to select as entry points: 

Analyzing /RedisGraph/src/redisgraph.so ...
   1) AC_GetU32  
   2) AGPLN_AddAfter    
   3) AGPLN_AddBefore                
   ...
   122) CommandDispatch
   123) ConcurrentCmdCtx_KeepRedisCtx                             
   124) ConcurrentSearch_HandleRedisCommand

Please select APIs by entering their numbers or names:

3. Select an entry point for analysis:

Enter “122” or type “CommandDispatch“. Coderrect then continues to analyze the library with “CommandDispatch” as an entry point:

 - â–– [00m:24s] Loading IR From File          
EntryPoints:
CommandDispatch
 - ✔ [00m:24s] Loading IR From File                    
 - ✔ [00m:40s] Running Compiler Optimization Passes (Phase I)                                              
 - ✔ [00m:00s] Canonicalizing Loops                    
 - ✔ [00m:00s] Propagating Constants                     
 - ✔ [03m:09s] Running Compiler Optimization Passes (Phase II)                                               
 - ✔ [00m:00s] Running Pointer Analysis                        
 - ✔ [03m:32s] Building Static Happens-Before Graph                                    
 - ✔ [00m:10s] Detecting Races               
 - ✔ [03m:39s] Scanning for additional OpenMP Regions

4. Analysis results:

In a few minutes, Coderrect generates the following results:

----------------------The summary of races in redisgraph.so------------------

==== Found a race between: 
line 389, column 21 in graph/graphcontext.c AND line 385, column 31 in graph/graphcontext.c
Shared variable:
 at line 39 of module.c
 39|GraphContext **graphs_in_keyspace;  // Global array tracking all extant GraphContexts.
Thread 1:
 387| if(graphs_in_keyspace[i] == gc) return;
 388| }
>389| graphs_in_keyspace = array_append(graphs_in_keyspace, gc);
 390|}
 391|
>>>Stacktrace:
>>>CommandDispatch
>>>  GraphContext_Retrieve [commands/cmd_dispatcher.c:68]
>>>    _GraphContext_Create [graph/graphcontext.c:132]
>>>      GraphContext_RegisterWithModule [graph/graphcontext.c:103]
Thread 2:
 383|void GraphContext_RegisterWithModule(GraphContext *gc) {
 384| // See if the graph context is not already in the keyspace.
>385| uint graph_count = array_len(graphs_in_keyspace);
 386| for(uint i = 0; i < graph_count; i ++) {
 387| if(graphs_in_keyspace[i] == gc) return;
>>>Stacktrace:
>>>CommandDispatch
>>>  GraphContext_Retrieve [commands/cmd_dispatcher.c:68]
>>>    _GraphContext_Create [graph/graphcontext.c:132]
>>>      GraphContext_RegisterWithModule [graph/graphcontext.c:103]...

  7 shared data races
 22 OpenMP races

To check the race report, please open '/git/RedisGraph/.coderrect/report/index.html' in your browser

Coderrect detected 7 shared data races and 22 OpenMP races in total in RedisGraph. The full report can be viewed in the browser.

Analyze static/dynamic library code

How to detect races in library code without a “main” method? This tutorial showcases this feature with a simple RingBuffer library libringbuffer.a, available on github.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Checkout the code and run coderrect with the following commands:

$ git clone https://github.com/coderrect/tutorial.git
$ cd tutorial/ringbuffer && cmake .
$ coderrect -t make

Coderrect will detect two public APIs in this library

1) RingBuffer::Consume       2) RingBuffer::Publish    

Please select APIs by entering their numbers or names (e.g. 1,2,RingBuffer::Consume,RingBuffer::Publish): 

In the terminal, type 1,2,3 and press Enter, Coderrect will report a race:

==== Found a race between: 
line 24, column 13 in ringbuffer_lib.cpp AND line 31, column 13 in ringbuffer_lib.cpp
Thread 1: 
 22|            buffer_[write_pos_] = value;
 23|            write_pos_++;
>24|            available_++;
 25|            return true;
 26|        }
>>>Stack Trace:
>>>pthread_create
>>>  coderrect_cb.1
>>>    RingBuffer::Publish(int)
Thread 2: 
 29|    }
 30|    bool RingBuffer::Consume(int *r) {
>31|        if (available_ == 0) {
 32|            return false;
 33|        }
>>>Stack Trace:
>>>pthread_create
>>>  coderrect_cb.2
>>>    RingBuffer::Consume(int*)
detected 1 races in total.

Behind the scene, Coderrect “simulates” the library’s behavior that RingBuffer::Publish and RingBuffer::Consume can be executed simultaneously by multiple threads. Even though there exists no “main” function, Coderrect will create two concurrent threads to invoke each API and detects the race.

Configuring Entry Points 

The library contains two public APIs: “RingBuffer::Publish” and “RingBuffer::Consume”, which can be called by multiple threads on a shared RingBuffer. To detect races in this library, users can also specify these APIs as entry points in a configuration file “.coderrect.json“

// .coderrect.json
{
  "entryPoints": [
    "RingBuffer::Publish",
    "RingBuffer::Consume"
  ]
}

Note: the namespace “RingBuffer::” can be skipped when no ambiguity exists.

Place the above configuration file under where you run Coderrect. Then run coderrect -t make again. You will see that the same race is reported.

More Advanced Option

Even better, configuring the entry points is unnecessary, but these entry points can be discovered automatically by Coderrect with the option -racedetect.analyzeApi. 

$ coderrect -t -racedetect.analyzeApi make

You will see that the same race as before is reported without any configuration. 

Caveat: Coderrect currently does not infer if an API must be executed before another. If that’s the case, false positives may be generated. To avoid them, explicitly configure entry points. 

The full library code is shown below.

//"ringbuffer_lib.h"
#ifndef RINGBUFFER_LIB_H
#define RINGBUFFER_LIB_H

#include <cstddef>
#include <stdexcept>
    class RingBuffer {
    private:
        int *buffer_;
        size_t write_pos_;
        size_t available_;
        size_t capacity_;

    public:
        RingBuffer(size_t capacity);
        ~RingBuffer();

        bool Publish(int value);
        bool Consume(int *r);
    };

#endif //RINGBUFFER_LIB_H

//"ringbuffer_lib.cpp"
  
#include "ringbuffer_lib.h"

    RingBuffer::RingBuffer(size_t capacity) : capacity_(capacity) {
        if (capacity == 0)
            throw std::invalid_argument("capacity must be greater than 0");

        buffer_ = new int[capacity];
        available_ = 0;
        write_pos_ = 0;
    }
    RingBuffer::~RingBuffer() {
        if (buffer_ != nullptr)
            delete[] buffer_;
    }
    bool RingBuffer::Publish(int value) {
        if (available_ < capacity_) {
            if (write_pos_ >= capacity_) {
                write_pos_ = 0;
            }
            buffer_[write_pos_] = value;
            write_pos_++;
            available_++;
            return true;
        }

        return false;
    }
    bool RingBuffer::Consume(int *r) {
        if (available_ == 0) {
            return false;
        }
        int next_slot = write_pos_ - available_;
        if (next_slot < 0) {
            next_slot += capacity_;
        }
        *r = buffer_[next_slot];
        available_--;
        return true;
    }

Here is the file “CMakeLists.txt“:

cmake_minimum_required(VERSION 3.13)
project(ringbuffer)
set(CMAKE_CXX_STANDARD 11)
find_package(Threads)
add_library(ringbuffer STATIC
            ringbuffer_lib.cpp
            ringbuffer_lib.h)

Coderrect Fast and Exhaust Modes

At different phases of the software development, developers may have different time and resource constraints. When editing code in an IDE, users may like to run Coderrect repeatedly and see the results as fast as possible. When testing code before a release, users may want to run Coderrect overnight on machines with large memory and core counts to exhaustively check all possible errors. 

Pay-as-you-go detection

To fit in different use scenarios, Coderrect offers pay-as-you-go detection: the more time you afford to run Coderrect, the better results, i.e., more code coverage and more precise results, you will likely get.

Coderrect has three modes: default, fast, and exhaust. These modes can be configured by the -mode option:

$ coderrect -mode=[default|fast|exhaust] 

Default mode aims to fit most uses. It exploits a balance between performance, code coverage and precision. The default mode is pretty fast and scalable: it can analyze over a million lines of C/C++ code such as the Linux kernel in less than ten minutes. 

Fast mode is optimized for speed. For large code base it can be many times faster than the default mode, even though it may possibly lose some analysis precision. At code development time, users may prefer to run Coderrect with “-mode=fast” to turn on the fast mode to reduce the waiting time. 

$ coderrect -mode=fast make

Exhaust mode is optimized for coverage and will exhaustively check all possible races under all code paths, all files and modules, as well as all call chains and dependencies. 

At nightly build time, you may prefer to run Coderrect with “-mode=exhaust” to find all possible race conditions as precise as possible when there is enough time to run (e.g., eight hours): 

$ coderrect -mode=exhaust make

Sample Performance Results on Redis

The following shows the performance of the fast mode on analyzing the Redis server:

The fast mode finishes in less than two minutes and detects 12 races:

The following shows the performance of the exhaust mode on analyzing the Redis server:

The exhaust mode finishes in around 75 mins and detects 22 races:

Blacklist and rank race conditions through configuration

How to config Coderrect to intentionally ignore certain race conditions or rank them by priority? This tutorial showcases this feature on an open-source project: memcached.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Run the following commands to clone the Git repository of memcached, install the libevent dependency, configure the build, and then run Coderrect:

$ git clone https://github.com/memcached/memcached.git
$ cd memcached
$ apt-get install libevent-dev autotools-dev automake
$ ./autogen.sh
$ ./configure
$ coderrect -t make

Detecting races

In a few seconds, you are expected to see a list of executables built by memcached:

The project creates multiple executables. Please select one from the list below
to detect races. 
In the future, you can specify the executable using the option "-e" if you know 
which ones you want to analyze. 
    coderrect -e your_executable_name1,your_executable_name2 your_build_command_line
 1) timedrun
 2) sizes
 3) memcached-debug
 4) memcached
 5) testapp

Select 3 to detect races in memcached-debug. Coderrect will report:

line 128, column 9 in assoc.c AND line 107, column 28 in assoc.c
Shared variable: 
hashpower at line 35 of assoc.c
 35|unsigned int hashpower = HASHPOWER_DEFAULT;
Thread 1: 
 126|        if (settings.verbose > 1)
 127|            fprintf(stderr, "Hash table expansion startingn");
>128|        hashpower++;
 129|        expanding = true;
 130|        expand_bucket = 0;
>>>Stack Trace:
>>>pthread_create [assoc.c:277]
>>>  assoc_maintenance_thread [assoc.c:277]
>>>    assoc_expand [assoc.c:256]
Thread 2: 
 105|
 106|    if (expanding &&
>107|        (oldbucket = (hv & hashmask(hashpower - 1))) >= expand_bucket)
 108|    {
 109|        pos = &old_hashtable[oldbucket];
>>>Stack Trace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    crawler_expired_eval [crawler.c:419]
>>>      do_item_unlink_nolock [crawler.c:219]
>>>        assoc_delete [items.c:531]
>>>          _hashitem_before [assoc.c:172]
...

...
...
...
detected 25 races in total.

So Coderrect detected 23 shared data races, 2 mismatched API issues, 4 TOCTOU. However, the races reported might be false positives.

Blacklisting races

To ignore race detections in certain part of the code that you don’t care about, you can configure Coderrect in three ways: 

1) ignore the function entirely through option “ignoreRacesInFunctions”,

2) ignore certain locations by file name and line number through option “ignoreRacesAtLocations”, and

3) ignore certain variables by their name through option “ignoreRaceVariables”.

All you need to do is to write a config file named coderrect.json

// .coderrect.json
{
  "ignoreRacesInFunctions": [
        "assoc_expand",
        "logger_create"
    ],
  "ignoreRacesAtLocations": [
        "items.c:1277",
        "extstore.c:493"
    ],
  "ignoreRaceVariables": [
        "crawlers",
        "wrap"
    ]
}

and put it under the path where you run Coderrect (In this case, it is the root path of memcached).

The config file above allows Coderrect to bypass all methods whose name matches “assoc_expand” or “logger_create”, ignore potential races in file “items.c” line “1277” and file “extstore.c” line “493“, and on global variables “crawlers” and “wrap“.

Now rerun Coderrect

$ make clean && coderrect -t make

Only 19 shared data races, 2 mismatched API issues, 3 TOCTOU were reported this time (and all of them are likely real true races):

Configure race priority

If you think certain races are more critical or less critical than others, you can configure them through the options “lowPriorityRaces“, “lowPriorityFiles“, and “highPriorityRaces“:

"highPriorityRaces": [
        "heads",
        "tails"
    ],
"lowPriorityRaces": [
        "stats*",
        "settings*"
    ]

The config above allows Coderrect to rank races whose variable’s names match “heads” and “tails” higher in the report, and those match “stats*”and “settings*” lower in the report. 

"lowPriorityFiles": [
        "*debug*"
    ],
"highPriorityFiles": [
        "*mem*"
    ]

The config above allows Coderrect to rank races whose file names match “*mem*” higher in the report, and those match “*debug*” lower in the report.

Can They Run in Parallel? A Must-Not-Run-In-Parallel Analysis

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

While locks and semaphores are frequently used to prevent data races in critical sections, developers also use standard synchronizations such as condition variables and barriers to prevent threads from running in parallel on a pair of functions. Coderrect performs a pretty precise “must-not-run-in-parallel” analysis that takes into consideration most common synchronization APIs such as pthread fork, join, wait, signal, broadcast, barrier, etc, as well as mutually exclusive branch conditions. More technical details can be found in a research paper.

If the user’s code relies on customized synchronizations to prevent threads from running in parallel, such as the once-only execution in the Open vSwitch OVS project. Coderrect may not recognize them and hence may report false warnings.

To address the issue, Coderrect supports user-specified not-run-in-parallel APIs through the configuration file .coderrect.json.

For example, adding the following to .coderrect.json will help Coderrect to recognize function1 and function2 cannot run in parallel. Thus Coderrect will not report data races between these two functions (as well as code called from them).

//.coderrect.json"
"notParallelFunctionPairs": {
    "function1" : "function2"
}

You can add multiple pairs of such functions and can also use wildcard (*) to match with any function. For example, bioInit and bioProcessBackgroundJobs are one pair of not-run-in-parallel functions,  ovsrcu_init_module and ovsrcu_init_module are another pair, meaning that ovsrcu_init_module can only be executed once and it cannot race with itself.

taosThreadToOpenNewFile and * are another pair, meaning that taosThreadToOpenNewFile can only be executed in a sequential environment and thus cannot race with any other function.

"notParallelFunctionPairs": {
    "bioInit":"bioProcessBackgroundJobs",
    "ovsrcu_init_module":"ovsrcu_init_module",
    "taosThreadToOpenNewFile" : "*"
}

Note that for each pair declared in notParallelFunctionPairs, the order of the specified APIs are interchangeable.

Coderrect and Jenkins Integration (Using Jenkinsfile)

This tutorial shows how to integrate Coderrect with Jenkins, the most popular CI/CD platform.

Coderrect is a static analysis tool for C/C++ code. Unlike popular tools such as Cppcheck and Clang static analyzer, one of the powerful features of Coderrect is the detection of concurrency races caused by code using explicit multi-thread APIs (e.g. PThread, std::thread) or implicit parallel APIs (e.g. OpenMP). 

Besides running Coderrect to scan the local source code via the command line, you can easily integrate it with Jenkins to include it in your CI/CD pipeline so that Coderrect can be triggered automatically by a pull request, a code check-in, or a scheduled nightly build.

Overview

The diagram below illustrates the interaction among components in Coderrect-Jenkins integration.

When a build is scheduled Jenkins master asks Jenkins agents to prepare the source code (usually pull the latest code from a source control system), build the project, test and analysis the program. 

Upon receiving the request from Jenkins Master, the Jenkins agent runs Coderrect installed locally to analyze the code and report discoveries back to the master. 

Install Coderrect

The first step is to install the Coderrect package to the Jenkins agent machine. Note that Coderrect just supports Linux for now. 

1. Download the latest version of Coderrect from here (the source can be found here).
2. Unpack the downloaded tarball.
3. Add /path/to/coderrect-installation/bin into PATH.

Install Coderrect Jenkins Plugin

You can download Coderrect Jenkins plugin here. Then go to the “Advanced” tab of “Manage Plugins” page.

For “Upload Plugin”, click “Choose File” to select the downloaded coderrect.hpi. 

Create a multi-branch project

Now we can create a multi-branch project in Jenkins.

Go to Jenkins portal and click “New Item“.

Enter the project name and select “Multibranch Pipeline” 

Set up the source control and build information. Make sure you use Jenkinsfile to control the build pipeline.

Create Jenkinsfile

We already created a project in Jenkins. To let Jenkins know how to build, test and analyze the code, we have to create a file called Jenkinsfile under the root directory of your repository.

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh "coderrect make"
            }
        }

        stage('Analysis') {
            steps {
                publishCoderrect ""
            }
        }
    }
}

The Jenkinsfile above defines a pipeline with two stages. The “build” stage builds the project and runs coderrect to analyze the generated binary. The “analysis” stage reports the analysis results to the master.

You may be aware that the command “publishCoderrect” has a parameter whose value is “” in our example. What is it? It is the relative path (to the project root directory) where you executes “coderrect”. We execute “coderrect” immediately under the project root so that we specify its value to “”.

If we execute “coderrect” under a directory rather than the project root we need to set up this parameter correctly. For example, the following figure shows typical build steps for a cmake-based project:

# we are under the project root directory
mkdir build
cd build
cmake ..
coderrect make

In this case you execute “coderrect” under project-root/build. You need to set the build directory to “build”.

#pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh "mkdir build && cd build && cmake .. && coderrect make"
            }
        }

        stage('Analysis') {
            steps {
                     publishCoderrect "build"
            }
        }
    }
}

Now we are done! We can go to the project dashboard and manually trigger a build.

Trigger a Build

Coderrect adds a section into the build summary page. The section shows the number of races in different categories for this build and the previous build. For example, the demo project above has 13 data races.

You can click the “Coderrect” item in the navigation bar to see details.

The detail report has three views – full, new and solved. The “full” view shows all races found from the current build. The “new” view shows just races introduced by the new committed code since previous build. The “solved” view shows races fixed by the new code.

I hope you enjoy this post. Please send to feedback@coderrect.com if you have any feedback.

Coderrect and Jenkins Integration (Using freestyle project)

This tutorial shows how to integrate Coderrect with Jenkins, the most popular CI/CD platform.

Coderrect is a static analysis tool for C/C++ code. Unlike popular tools such as Cppcheck and Clang static analyzer, one of the powerful features of Coderrect is the detection of concurrency races caused by code using explicit multi-thread APIs (e.g. PThread, std::thread) or implicit parallel APIs (e.g. OpenMP). 

Besides running Coderrect to scan the local source code via the command line, you can easily integrate it with Jenkins to include it in your CI/CD pipeline so that Coderrect can be triggered automatically by a pull request, a code check-in, or a scheduled nightly build.

Overview

The diagram below illustrates the interaction among components in Coderrect-Jenkins integration.

When a build is scheduled Jenkins master asks Jenkins agents to prepare the source code (usually pull the latest code from a source control system), build the project, test and analysis the program. 

Upon receiving the request from Jenkins Master, the Jenkins agent runs Coderrect installed locally to analyze the code and report discoveries back to the master. 

Install Coderrect

The first step is to install the Coderrect package to the Jenkins agent machine. Note that Coderrect just supports Linux for now. 

1. Download the latest version of Coderrect from here (the source can be found here).
2. Unpack the downloaded tarball.
3. Add /path/to/coderrect-installation/bin into PATH.

Install Coderrect Jenkins Plugin

You can download Coderrect Jenkins plugin file here. Then go to the “Advanced” tab of “Manage Plugins” page.

For “Upload Plugin”, click “Choose File” to select the downloaded coderrect.hpi. 

Create a Freestyle project

Now we can create a freestyle project in Jenkins.

Go to Jenkins portal and click “New Item“.

Enter the project name (TDEngine here) and select “Freestyle project” 

Set up the source control and build information. For the build stage, I create the following script commands

The last step “coderrect -e taosd -t make” tells coderrect to

• Execute “make” to build binaries
• Analyze an executable called “taosd” to detect races
• Generate a terminal-based race report besides the default HTML-based report

Set up the post-build stage by choosing “Publish the race report to Jenkins”

Since we run “coderrect” under the project_root/build, we specify a relative path “build” so that coderrect knows where to find the race report.

You are ready to go ffter you click “Save”!

Trigger a Build

Coderrect adds a section into the build summary page. The section shows the number of races in different categories for this build and the previous build. For example, the demo project above has 13 data races.

You can click the “Coderrect” item in the navigation bar to see detail report.

The detail report has three views – full, new and solved. The “full” view shows all races found from the current build. The “new” view shows just races introduced by the new committed code since previous build. The “solved” view shows races fixed by the new code.

I hope you enjoy this post. Please send to feedback@coderrect.com if you have any feedback.

Github CI/CD Integration

If you are already running a CI/CD service on Github, it is easy to add Coderrect to your workflows. In a nutshell, add the following to .github/workflows/ci.yml

- name: coderrect setup
        run: |
            wget https://public-installer-pkg.s3.us-east-2.amazonaws.com/coderrect-linux-1.1.3.tar.gz
            tar zxf coderrect-linux-1.1.3.tar.gz
            echo "$PWD/coderrect-linux-1.1.3/bin" >> $GITHUB_PATH
- name: make
       run: coderrect -analyzeAllBinaries make
- name: coderrect check
        run: coderrect -publishResults

Background

Continuous Integration/Continuous Delivery (CI/CD) is a common practice now for developing software. In CI, typically tests and checks are run against every pull request and code commit to ensure the new code changes do not break anything or introduce new bugs. CI makes it easier to fix bugs quickly and often.

If your code is hosted on Github, you can create custom CI/CD workflows directly in your GitHub repository with GitHub Actions. All you need is to add a .yml file under .github/workflows/ of your Github repo.

As an example, Redis has a ci.yml file with the following content:

name: CI
on: [push, pull_request]
jobs:
    test-ubuntu-latest:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v2
          - name: make
            run: make
          - name: test
            run: |
                sudo apt-get install tcl8.5
               ./runtest --verbose
         - name: module api test
            run: ./runtest-moduleapi --verbose

With the configuration above, Github CI will build and test Redis upon every code push and pull request. If the build is successful and the tests are all passed, Github will show a green tick associated with the commit. 

Otherwise, if the build or any test fails, you will see a red cross: 

Integrating Coderrect into Github CI

To run coderrect in the Github CI, in ci.yml, simply add the following three steps:

# Step 1: download coderrect and add to path
- name: setup coderrect
        run: |
           wget https://public-installer-pkg.s3.us-east-2.amazonaws.com/coderrect-linux-0.8.0.tar.gz
           tar zxf coderrect-linux-0.8.0.tar.gz
           echo "::add-path::$PWD/coderrect-linux-0.8.0/bin"

# Step 2: build and run coderrect
- name: make
        run: coderrect -analyzeAllBinaries make

# Step 3: coderrect results
- name: coderrect check
        run: coderrect -publishResults

In Step 1, coderrect is setup; 

In Step 2, the flag -analyzeAllBinaries tells Coderrect to check all the build targets;

In Step 3, if any race is detected the results will be reported in the CI terminal. A red cross sign will be shown, indicating potential race hazards found by Coderrect: 

To see the detailed race information, just click the arrow sign to expand it: 

Cache the downloaded Coderrect

Coderrect only needs to be downloaded once for Github CI. To cache the downloaded Coderrect package, simply add the following to ci.yml.

- name: cache coderrect
    id: cache-coderrect
    uses: actions/cache@v2
    with:
        path: coderrect-linux-0.8.0
        key: ${{ runner.os }}-coderrect

- name: setup coderrect
    if: steps.cache-coderrect.outputs.cache-hit != 'true'

A full example is available at here.

Installing Coderrect with Spack

Spack is an increasingly popular package manager for Linux and macOS that eliminates the difficulties of managing complex software dependencies. Spack is especially popular in the high performance computing community. Spack won the prestigious R&D 100 Award in 2019 and is the official deployment tool of the ExaScale Computing Project which is tasked finding the tools and systems to support the next generation of ExaScale super computers in the U.S. Spack can even be used to set up a cloud-HPC cluster in AWS.

Prerequisites

This tutorial assumes you already have Spack installed. If you are new to Spack, we recommend following Spack’s Getting Started Guide.

Installing Coderrect with Spack

First download and add the Coderrect Spack repository to your local machine.

$ wget https://github.com/coderrect-inc/coderrect-spack/archive/master.zip
$ unzip master.zip && rm master.zip
$ mv coderrect-spack-master coderrect-spack
$ spack repo add coderrect-spack

Once the repo is registered, Spack makes its easy to install and load the Coderrect tool.

$ spack install coderrect
$ spack load coderrect

Coderrect is now installed and ready to use. Spack really does make things simple!

Check out the Coderrect Quick Start page or some of the other tutorials to see examples of how to use the Coderrect tool.

CODERRECT SOFTWARE EVALUATION LICENSE AGREEMENT

IMPORTANT-READ CAREFULLY: This Single User License Agreement (“Agreement”) is a legal agreement between you (either an individual person or a single legal entity, who will be referred to in this Agreement as “You”) and Coderrect Inc. for the Coderrect software that accompanies this Agreement, including any associated media, printed materials and electronic documentation (collectively “Software”). The Software also includes any software updates, add-on components, web services and/or supplements that Coderrect Inc. may provide to you or make available to you after the date you obtain your initial copy of the Software to the extent that such items are not accompanied by a separate license agreement or terms of use. By installing, copying, downloading, accessing or otherwise using the Software, You agree to be bound by the terms of this Agreement. If you do not agree to the terms of this Agreement, do not install, access or use the Software.

GENERAL TERMS AND CONDITIONS

Coderrect Inc. grants you ONE (1) non-exclusive and non-transferable single user license for the internal use only of the Software. This license entitles the owner to install the Software on a single computer. You will not otherwise use or copy, translate, modify, adapt, decompile, disassemble or reverse engineer the Software. You agree not to use the Software in violation of any law, statute, ordinance or other regulation (including export control and unfair competition laws) or any obligation to which you are bound. 

You agree to comply with all applicable laws and regulations regarding your use of the Software. You agree to indemnify Coderrect Inc. from and against any liability that Coderrect Inc. may incur arising from your use of the Software.

You may use the Software solely to evaluate and test the Software pursuant to the terms of this Agreement during the evaluation period. Since this is evaluation Software, portions of the full use version of the Software may be withheld or unusable and use of the Software may require accessing portions of the Software remotely through the Internet.

OWNERSHIP

All patents, copyrights, trade secrets and other proprietary rights in or related to the Software are and will remain the exclusive property of Coderrect Inc., whether or not specifically recognized or perfected under the laws of the country where the Software is located. You hereby assign and agree to assign any and all rights you may have or acquire in or to the Software to Coderrect Inc. for no additional consideration. 

You will not take any action that jeopardizes Coderrect Inc.’s or its licensors’ proprietary rights or acquire any rights in the Software. Coderrect Inc. or its designee will own all rights in any copy, translation, modification, adaptation or derivation of the Software, including any improvement or development thereof. At Coderrect Inc.’s request, you will execute or obtain the execution of any instrument that may be appropriate to assign these rights to Coderrect Software or its designee or to perfect these rights in Coderrect Inc.’s or its designee’s name.

CONFIDENTIALITY

(a) Confidentiality. You acknowledge that the Software is and incorporates confidential and proprietary information developed, acquired by or licensed to Coderrect Inc.. You will take all reasonable precautions necessary to safeguard the confidentiality of the Software, and will not disclose any information about the Software or the Software evaluations or reports to any other person without Coderrect Inc.’s prior written permission. You will not allow the removal or defacement of any confidentiality or proprietary notice placed on the Software. The placement of a copyright notice on the Software will not constitute publication or otherwise impair its confidential nature.

(b) Unauthorized Use or Disclosure. You acknowledge that any unauthorized use or disclosure of the Software will cause irreparable harm to Coderrect Inc. and its licensors. If an unauthorized use or disclosure occurs, you will take all steps that are necessary to recover the Software and to prevent its subsequent unauthorized use or dissemination.

(c) Limitation. You will have no confidentiality obligation with respect to any portion of the Software that (i) You lawfully obtained from a third party under no obligation of confidentiality or (ii) became available to the public other than as a result of Your act or omission.

DISCLAIMER OF WARRANTY

UNLESS SPECIFIED IN THIS AGREEMENT, ALL EXPRESS OR IMPLIED CONDITIONS, REPRESENTATIONS AND WARRANTIES, INCLUDING ANY IMPLIED WARRANTY OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE, OR NON-INFRINGEMENT, ARE DISCLAIMED, EXCEPT TO THE EXTENT THAT THESE DISCLAIMERS ARE HELD TO BE LEGALLY INVALID.

 DISCLAIMER OF DAMAGES

TO THE EXTENT NOT PROHIBITED BY LAW, IN NO EVENT WILL CODERRECT INC. OR ITS LICENSORS BE LIABLE FOR ANY LOSS IN REVENUE, PROFIT OR DATA, OR FOR SPECIAL, INDIRECT, CONSEQUENTIAL, INCIDENTAL OR PUNITIVE DAMAGES, HOWEVER CAUSED REGARDLESS OF THE THEORY OF LIABILITY, ARISING OUT OF OR RELATED TO THE USE OF OR INABILITY TO USE THE SOFTWARE, EVEN IF CODERRECT INC. HAS BEEN ADVISED OF THE POSSIBILITY OF SUCH DAMAGES. IN NO EVENT WILL CODERRECT INC.’S LIABILITY TO YOU, WHETHER IN CONTRACT, TORT (INCLUDING NEGLIGENCE), OR OTHERWISE, EXCEED THE AMOUNT PAID BY YOU FOR SOFTWARE UNDER THIS AGREEMENT. THE FOREGOING LIMITATIONS WILL APPLY EVEN IF THE ABOVE STATED WARRANTY FAILS OF ITS ESSENTIAL PURPOSE.

TERMINATION

This Agreement is effective until terminated. You may terminate this Agreement at any time by destroying all copies of the Software. This Agreement will terminate immediately without notice from Coderrect Inc. if you fail to comply with any provision of this Agreement. Upon Termination, you must destroy all copies of Software. Upon termination, you agree to promptly de-install and delete all copies of the Software in Your possession and discontinue any further use of the Software. The parties’ rights and obligations will survive the termination of this Agreement.

MISCELLANEOUS

You and Coderrect Inc. are independent parties. Nothing in this Agreement shall be construed as making you an employee, agent or legal representative of Coderrect Inc.. 

This Agreement constitutes the complete and entire statement of all terms, conditions and representations of the agreement between you and Coderrect Inc. with respect to the Software.

If any provision of this Agreement is held to be unenforceable, This Agreement will remain in effect with the provision omitted, unless omission would frustrate the intent of the parties, in which case this Agreement will immediately terminate.

This Agreement is the entire agreement between you and Coderrect Inc. relating to its subject matter. It supersedes all prior or contemporaneous oral or written communications, proposals, representations and warranties and prevails over any conflicting or additional terms of any quote, order, acknowledgment, or other communication between the parties relating to its subject matter during the term of this Agreement. No modification of this Agreement will be binding, unless in writing and signed by an authorized representative of each party.

For inquiries please contact:

Coderrect Inc.
3206 Longmire
College Station, TX 77845, USA

V0.9.1

• The Coderrect team announces a new version of Coderrect, a super fast, precise and scalable static race detector. Coderrect software can be downloaded from the download page.
• Coderrect software is under an evaluation license.
• For more information about Coderrect, start with the product page. If you have questions or comments, please send us an email.

• Release 0.9.1 includes some small refinements to improve race detection accuracy

What’s New in V0.9.0

• Better Fortran module support and syntax compatibility.
• Incremental race comparison.
• Configuration can include other configuration files using the keyword ‘include’.
• Configuration supports ‘@’ to merge values of the same key from multiple configuration files.
• Function-oriented thread API profile (tutorial).
• Upgrade to clang/llvm-10.
• More complete call stack for inlined functions
• Improved detection accuracy on C++ vector operations
• Reentrant deadlock detection

Minimum System Requirements

Linux-based OS:

• Ubuntu 14+
• CentOS 7+
• Debian 8+

4GB+ RAM (recommended)

Previous Release Notes

V0.8.1

The latest 0.8.1 release allows the Coderrect Scanner to handle a wider variety of build environments.
Although the Coderrect Scanner uses clang internally, release 0.8.1 introduces preliminary support for analyzing projects that are built by other compilers, like icc or icpc, that use flags not recognized by clang.

V0.8.0

• Support Fortran (90, 95, and 2003), see tutorial.
• Support more OpenMP (4.5 and 5) features including sections, tasks and taskgroup.
• Support static array bound verification.
• Optimize detection performance and accuracy for both Pthreads and OpenMP races.
• Allow the user to register open-source projects in our OpenScan page.

V0.7.0

• Detect multithreaded TOCTOU issues (see tutorial).
• Better OpenMP array indirect analysis.
• Allow the user to configure customized lock-unlock functions (see the tutorial)
• Allow the user to configure non-parallel functions (see the tutorial).
• Cleanup configurations (see the complete configuration manual).
• Allow Coderrect to continue analysis even when it encounters build errors.
• Optimize detection performance.

V0.6.0

• Support C++11 std::thread APIs.
• Support C++ standard container APIs.
• Add Exhaust mode (option “-mode=exhaust”) to exhaustively detect all possible races aiming for full coverage (checkout the tutorial).
• Add Fast mode (option “-mode=fast”) to detect races as fast as possible aiming for faster detection speed (checkout the tutorial). 
• Support Ubuntu 20.04 and other Linux distributions shipped with libtinfo.6. (for older versions the libtinfo issue has been resolved)
• Support Github CI/CD integration (experimental, checkout the tutorial).
• Support Jenkins CI/CD integration (experimental, checkout the tutorial).
• The race report shows the data structure field involved in a data race.
• The race report records the configuration settings.
• Support user specified lock/unlock APIs (checkout the tutorial)
• Support analyzing multiple binaries generated by a project (checkout the tutorial).

Download a PDF version here.

Overview

Coderrect is a fast static detector for C/C++/Fortran/OpenMP multi-threaded bugs.

Coderrect identifies critical race conditions in high-performance, highly-concurrent C/C++/Fortran code. Precise location and call stack trace leading to the race are provided to help developers understand how to fix them quickly. Coderrect seamlessly integrates automated testing into your CI/CD pipelines and supports your existing development tools and workflows.

Key features

Fast and accurate analysis

• Coderrect’s “analysis without running” feature enables developers to detect race conditions without running the program. Simply prepend “coderrect” to your project build command line, and Coderrect will automatically analyze the code.
• Coderrect supports three modes: fast, normal, and full. Fast mode allows developers to analyze the project with hundreds of lines of code in tens of seconds. Full mode allows exhaustively checking all execution paths for most subtle issues.

Comprehensive reporting

• Coderrect provides developers all the information they need to understand how to fix the issues identified, including detailed descriptions, categories, shared variables, code snippets, and stack traces. 

• Developers can choose to view a terminal-based report or to generate a HTML-based detail report.

Software development life cycle integrations

• Coderrect supports native integration for Github.
• Coderrect provides plugins and integration for Jenkins.

OpenMP Support

• Specially designed analysis for OpenMP regions.
• Support for common OpenMP features:
  • parallel, for, barrier, master, single, reduction, atomic, critical, target, and more.
• Specialized array index analysis for detecting data races on arrays and matrices.
• Support for OpenMP offloaded to GPUs through the target construct.

Technical Specification

Supported Compilers

• Clang
• Intel C++
• GNU GCC/G++
• GFortran/G77
• IFORT
• Flang

Critical checks

• Atomicity violations
• Order violations
• Deadlocks
• Data races
• Time of check to time of use (TOCTOU)

Supported Operating Systems

• Ubuntu 14.04+
• CentOS 7+
• Redhat 7+
• Debian 8+
• Arch

SDLC native integrations

• Github
• Jenkins 2+

Configurations

Configurations are set in a configuration file named .coderrect.json. This configuration file should be placed in the directory where Coderrect is called.

Each configuration setting can only appear once in a configuration file, for example, you can NOT have multiple skipFunctions sections in a configuration file. 

Command Line Options (by function groups)

Configurations to control the race detection algorithm

Skip Functions

Config Coderrect to skip specified functions during race analysis.

"skipFunctions":[
    "FunctionName*",
    "FunctionOther"
] 

Check out a tutorial uses this configuration here

Indirect Functions

Config Coderrect to analyze all possible targets called in specified functions.

"indirectFunctions":[
    "FunctionName*",
    "FunctionOther"
]

Check out a tutorial uses this configuration here

Ignore Races in Functions

Config Coderrect to skip analysis specified functions.

"ignoreRacesInFunctions":[
    "std::*"
    "FunctionName*",
    "FunctionName2"
]

This configuration by default is set to skip all functions matching std::*.

Check out a tutorial uses this configuration here

Ignore Races at Locations

Config Coderrect to skip analysis on specified locations, i.e. filename and line numbers.

"ignoreRacesAtLocations": [
    "items.c:1277",
    "extstore.c:493"
],

Check out a tutorial uses this configuration here

Ignore Races by Variable Name

Config Coderrect to skip analysis on specified variables.

"ignoreRaceVariables":[
    "VariableName*",
    "VariableOtherName"
]

Check out a tutorial uses this configuration here

Set Low Priority Files

Races detected in specified files are set to a lower priority in the report, and shown lower in the report.

"lowPriorityFiles":[
    "FileName*",
    "FileNameOther"
]

Check out a tutorial uses this configuration here

Set High Priority Files

Races detected in specified files are set to a higher priority in the report, and shown lower in the report.

"highPriorityFiles":[
    "FileName*",
    "FunctionNameOther"
]

Check out a tutorial uses this configuration here

Set Low Priority Races

Races detected in specified variables are set to a lower priority in the report, and shown lower in the report.

"lowPriorityRaces":[
    "VariableName*",
    "VariableNameOther"
]

Check out a tutorial uses this configuration here

Set High Priority Races

Races detected in specified variables are set to a higher priority in the report, and shown lower in the report.

"highPriorityRaces":[
    "VariableName*",
    "VariableNameOther"
]

Check out a tutorial uses this configuration here

Mark Heap Allocation Functions

Specify functions to be treated as allocating memory.

"heapAllocFunctions":[
    "je_malloc",
    "je_calloc",
    "je_mallocx",
    "rm_malloc",
    "rm_calloc"
]

The setting show in the above example is the default setting.

Set Entry Points to Analyze Libraries

Specify entry points for libraries without a main function.

"openlib": {
    "entryPoints":[
        "FunctionName*",
        "FunctionNameOther"
    ]
}

Check out a tutorial uses this configuration here

Specify Custom Lock/Unlock APIs

Specify a list of custom lock/unlock APIs so that Coderrect can make corrrect reasoning. 

"lockUnlockFunctions": {
    "MySpinLock": "MySpinUnlock",
    "MyMutexLock": "MyMutexUnlock"
}

Check out a tutorial uses this configuration here

The default global configuration file

You can find the default global configuration file “coderrect.json” under $installDir/conf.

{
  "logger": {
    "level": "debug",
    "logFolder": "logs",
    "maximumLogFileCount": 8,
    "maximumLogFileSize": 128,
    "toStderr": false
  },

  "report": {
    "enableTerminal": false,
    "//ouputDir": "please remove // and set the right path here"
  },

  "cleanBuild":                   false,
  "analyzeAllBinaries":           false,
  "continueIfBuildErrorPresents": false,
  "//bcPaths":                    "Remove // and specify a list of comma-separated paths",
  "//analyzeBinaries":            "Remove // and specify a list of comma-separated binary names",

  "publishResults":               false,

  "mode":                         "normal",
  "enableOpenMP":                 true,
  "enableOrderViolation":         true,
  "enableAtomicityViolation":     true,
  "enableMissingAPI":             true,
  "enableFilter":                 true,
  "enableLockSet":                true,
  "enableLoopUnroll":             true,

  "raceLimit":                    25,

  "skipFunctions":                [],
  "indirectFunctions":            [],
  "ignoreRacesInFunctions":       [
        "std::*"
  ],
  "ignoreRacesAtLocations":       [],
  "ignoreRaceVariables":          [],
  "lowPriorityFiles":             [],
  "lowPriorityRaces":             [],
  "highPriorityRaces":            [],
  "heapAllocFunctions":           [
     "je_malloc",
     "je_calloc",
     "je_mallocx",
     "rm_malloc",
     "rm_calloc"
  ],
  "lockUnlockFunctions":          {},
  "notParallelFunctionPairs":     {}

  "highPriorityFiles":            [],
  "lowPriorityFiles":             [],
  "highPriorityRaces":            [],
  "lowPriorityRaces":             [],

  "openlib": {
    "entryPoints":                [],
    "limit":                      999,
  }
}

• What’s Coderrect?

Coderrect is a fast and scalable tool that finds race conditions in complex software. 

• What does Coderrect mean, and how do you pronounce it?

Coderrect is a created word by combining Code and Correct.  It simply means code-correct. Coderrect is pronounced code-rect.

• Is Coderrect stable?

Coderrect is still a development release, but getting more stable.

• What are pre-requisites to run Coderrect successfully on my project?

Coderrect requires Linux-based OS, such as Ubuntu 14+, and CentOS 7+.  

To run Coderrect, you will need to compile your code successfully without Coderrect. For example, running make/cmake/Ninja/Bazel will build your project without errors.

• How does Coderrect work internally?

Coderrect generates an intermediate representation of your source code in the form of LLVM bitcode (BC) files and then performs sophisticated static analyses based on them to find potential race conditions. 

Coderrect produces a single BC file for each build binary (an executable, a static library, or a shared library).

• Does Coderrect change my build binaries?

No. Coderrect intercepts your build commands but will not change your build targets. Running Coderrect will generate exactly the same build binaries as w/o Coderrect. 

• What source languages are supported?

Coderrect supports most common C/C++ compilers (GCC/Clang/ICC). However, internally Coderrect will use clang to generate LLVM bitcode. So, it may not work if your code does not compile with clang, here is an example showing the difference between clang and gcc.

Coderrect also supports Fortran compilers including Flang/GFortran/IFORT.

• What compilers are supported?

Coderrect has full support for C and C++ source code through GCC/Clang/ICC, and Fortran source code through Flang/GFortran/IFORT.

• Where can I find the output of the code scan results?

Besides the console output, there is an HTML file named index.html generated under ./report directory by default. You can specify the location of the report through “-o <directory>” option on the command line.

• What license is Coderrect under?

Coderrect is currently under Evaluation License, and is subject to change in the future.

• I think that I have found a bug, what should I do?

Please contact us (contact@coderrect.com).

Updated 09/30/2020