Detect OpenMP Races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded C program written with OpenMP.

Prerequisites

• This tutorial assumes you have successfully installed the Coderrect software following the quick start.

• Our sample code relies on OpenMP to achieve parallelism. You will need a compiler that supports OpenMP. We will be using gcc, but clang and other modern alternatives also have OpenMP support. 

Detect a race in pi.c

We will start by detecting a race in a small single file program pi.c. The program is designed to compute pi=3.1415926…. We provide this example program in the Examples directory under Coderrect installation, or you can copy the code below to your system.

//pi.c
#include <omp.h>
#include <stdio.h>

#define N 1000000000

int main () {

    double delta = 1.0/(double) N;
    int MAX_THREADS = omp_get_max_threads();
    // Compute parallel compute times for 1-MAX_THREADS
    for (int j=1; j<= MAX_THREADS; j++) {

        printf(" running on %d threads: ", j);
        omp_set_num_threads(j);

        double sum = 0.0;
        double start = omp_get_wtime();

        #pragma omp parallel for //reduction(+:sum)
        for (int i=0; i < N; i++) {
            double x = (i+0.5)*delta;
            sum += 4.0 / (1.0+x*x);
        }

        // Out of the parallel region, finalize computation
        double pi = delta * sum;
        double time_lapse = omp_get_wtime() - start;
        printf("PI = %.12g computed in %.4g secondsn", pi, time_lapse);
    }
}

Check that pi.c can be compiled and run with the following commands

gcc -fopenmp pi.c -o pi
./pi

You should see output that looks something like:

running on 1 threads: PI = 3.141592653589971 computed in 12.84 seconds
running on 2 threads: PI = 3.141593993623682 computed in 6.928 seconds
running on 3 threads: PI = 3.141594228301372 computed in 7.741 seconds
running on 4 threads: PI = 3.141595112697573 computed in 8.376 seconds

As you can see from the results, running on different number of threads generated different values of PI, indicating the existence of a concurrency bug.

Run Coderrect

The easiest way to run the tool is by passing the build command to coderrect:

coderrect -t gcc -fopenmp pi.c

Remember, the command to build pi.c was gcc -fopenmp pi.c.

-t switch is used to generate a quick summary report in terminal.

Calling coderrect in this way ensures all the required compilation flags can be passed on the command line. For a project using a build system such as make, coderrect tool can be called with the same build command used to build the project. For an example: checkout out detecting races in a Makefile-based project.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in a browser.

Terminal Report

The terminal report for pi.c should look something like the following:

==== Found a race between: 
line 22, column 13 in test.c AND line 22, column 17 in test.c
Shared variable:
 at line 16 of test.c
 16|        double sum = 0.0;
Thread 1:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }
 24|
>>>Stacktrace:
Thread 2:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }
 24|
>>>Stacktrace:

                 1 OpenMP races

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 22, column 13 in test.c AND line 22, column 17 in test.c

Next the report shows the variable name and location on which the race occurs.

Shared variable:
 at line 16 of test.c
 16|        double sum = 0.0;

This shows that the race occurs on the variable sum declared on line 16.

Going and finding the race location in the code may be a little tedious so the report also shows a preview of the file at that location.

Thread 1:
 20|        for (int i=0; i < N; i++) {
 21|            double x = (i+0.5)*delta;
>22|            sum += 4.0 / (1.0+x*x);
 23|        }

The code snippet shows that this racing access is a write to sum as part of an OpenMP parallel for loop.

Taking a closer look at the source code we can see the root cause is the commented out “reduction”. 

#pragma omp parallel for //reduction(+:sum)

Un-commenting reduction(+:sum) removes the data race on sum and allows the program to calculate pi correctly.

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

We can also save the race report to a directory specified via the command option-o <dir>. 

coderrect -o report gcc -fopenmp pi.c

This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Detect pthread races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded C++ program written with the POSIX threads (Pthreads) library.

coderrect g++ pthread-race.cc -lpthread

Prerequisites

This tutorial assumes you have successfully installed the Coderrect software following the quick start.

Detecting races in pthread-race.cc

Our sample code pthread-race.cc starts two children threads in the main method. Each child thread increments a shared variable “x” by one, but there is a race.

Copy the source for pthread-test.cc below to your system.

#include <pthread.h>

#include <cstdlib>
#include <iostream>

using namespace std;
int x = 0;

void *PrintHello(void *threadid) {
    long tid;
    tid = (long)threadid;
    x++;
    cout << "Hello World! Thread ID, " << tid << endl;
    pthread_exit(NULL);
}

pthread_t load_data_in_thread(long id) {
    pthread_t thread;
    void *arg = (void *)id;
    int rc = pthread_create(&thread, NULL, PrintHello, arg);
    if (rc) {
        cout << "Error:unable to create thread," << rc << endl;
        exit(-1);
    }
    return thread;
}

int main() {
    pthread_t thread1, thread2;

    cout << "main() : creating thread 1 " << endl;
    thread1 = load_data_in_thread(1);
    cout << "main() : creating thread 2 " << endl;
    thread2 = load_data_in_thread(2);
    pthread_join(thread1, 0);
    // pthread_join(thread2,0);
    cout << "Final value of x: " << x << endl;
}

Check that pthread-race.cc can be compiled and run with the following commands

g++ pthread-race.cc -lpthread
./a.out

If run ./a.out multiple times, you should see output that looks something like:

main() : creating thread 1 
main() : creating thread 2 
Hello World! Thread ID, 1
Final value of x: 1
Hello World! Thread ID, 2

Run Coderrect

The easiest way to get started on a single file is to run

coderrect -t g++ pthread-race.cc -lpthread

-t switch is used to generate a quick summary report in terminal, for more information on Coderrect configurations, please check out reference page.

This will automatically detect the races and report the follow in the terminal:

==== Found a race between: 
line 12, column 5 in pthread-race.cc  AND  line 38, column 41 in pthread-race.cc
Shared variable: 
 at line 7 of pthread-race.cc
 7|int x =0;
Thread 1: 
pthread-race.cc@12:5
Code snippet: 
 10| long tid;
 11|    tid = (long)threadid;
>12|    x++;
 13|    cout << "Hello World! Thread ID, " << tid << endl;
 14|    pthread_exit(NULL);
>>>Stack Trace:
>>>PrintHello(void*) [pthread-race.cc:20]
Thread 2: 
pthread-race.cc@38:41
Code snippet: 
 36|      //pthread_join(thread2,0);
 37|
>38|        cout << "Final value of x: " << x << endl;
 39|}
>>>Stack Trace:
>>>main()
>>>  std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) [pthread-race.cc:38]
>>>    std::char_traits<char>::length(char const*) [/usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/ostream:562]
detected 1 races in total.

2020/02/25 01:18:23 Generating the race report ...
To check the race report, please open './index.html' in your browser

Interpret the Results

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 12, column 5 in pthread-race.cc  AND  line 38, column 41 in pthread-race.cc

Next the report shows the name and location of the variable on which the race occurs.

Shared variable: 
 at line 7 of pthread-race.cc
 7|int x =0;

This shows that the race occurs on the variable x declared on line 7.

Next the tool reports information about the two unsynchronized accesses to x.
For each of the two accesses a location, code snippet, and stack trace is shown.

The location shows the file, line, and column of the access.

pthread-race.cc@12:5
pthread-race.cc@38:41

So the above access occurs in pthread-race.cc at line 12 column 5 and at line 38 column 41, respectively.

Going and finding this location in the code may be a little tedious so the report also shows a preview of the file at that location.

Code snippet: 
 10| long tid;
 11|    tid = (long)threadid;
>12|    x++;
 13|    cout &lt;&lt; "Hello World! Thread ID, " << tid << endl;
 14|    pthread_exit(NULL);

The code snippet shows that this access is an unsynchronized write to x in each child thread.

The last piece of information shown for each access is the stack trace.

>>>Stack Trace:
>>>PrintHello(void*) [pthread-race.cc:20]

>>>main()
>>> std::basic_ostream<char, std::char_traits<char> >& std::operator<<<std::char_traits<char> >(std::basic_ostream<char, std::char_traits<char> >&, char const*) [pthread-race.cc:38]
>>> std::char_traits<char>::length(char const*) [/usr/lib/gcc/x86_64-linux-gnu/7.4.0/../../../../include/c++/7.4.0/ostream:562]

The stack trace shows the call stack under which the racing access occurred.
Each line in the call stack shows the name of the function, and the location the function was called from.

In the example above the stack trace shows PrintHello being called from line 20 in pthread-race.cc.

HTML Report

The full report can be viewed in a browser.

Detect races in GPU kernels

This tutorial showcases Coderrect on detecting block-level and warp-level race hazards in GPU/CUDA kernels. Both examples are from NVIDIA’s official documentation: block_error.cu and warp_error.cu. Note that if you do not already have nvidia-cuda-toolkit installed on your machine you must use the command listed below.

$ sudo apt install nvidia-cuda-toolkit
$ coderrect -t nvcc block_error.cu

==== Found a race between: 
line 9, column 5 in block_error.cu AND line 14, column 25 in block_error.cu
Shared variable: 
smem at line 3 of block_error.cu
 3|__shared__ int smem[THREADS];
Thread 1: 
 7|{
 8|    int tx = threadIdx.x;
>9|    smem[tx] = data_in[tx] + tx;
 10|
 11|    if (tx == 0) {
>>>Stack Trace:
Thread 2: 
 12|        *sum_out = 0;
 13|        for (int i = 0; i < THREADS; ++i)
>14|            *sum_out += smem[i];
 15|    }
 16|}
>>>Stack Trace:
The OpenMP region this bug occurs:
/CUDA/benchmarks/t/block_error.cu
>27|    sumKernel<<<1, THREADS>>>(data_in, sum_out);
 28|    cudaDeviceSynchronize();
 29|
 30|    cudaFree(data_in);
 31|    cudaFree(sum_out);
 32|    return 0;
Gets called from:
>>>main
detected 1 races in total.
To check the race report, please open '/CUDA/benchmarks/t/.coderrect/report/index.html' in your browser

$ coderrect -t nvcc wrap_error.cu

==== Found a race between: 
line 12, column 5 in wrap_error.cu AND line 19, column 32 in wrap_error.cu
Shared variable: 
smem_first at line 5 of wrap_error.cu
 5|__shared__ int smem_first[THREADS];
Thread 1: 
 10|{
 11|    int tx = threadIdx.x;
>12|    smem_first[tx] = data_in[tx] + tx;
 13|    //__syncwarp();
 14|    if (tx % WARP_SIZE == 0) {
>>>Stack Trace:
Thread 2: 
 17|        smem_second[wx] = 0;
 18|        for (int i = 0; i < WARP_SIZE; ++i)
>19|            smem_second[wx] += smem_first[wx * WARP_SIZE + i];
 20|    }
 21|
>>>Stack Trace:
The OpenMP region this bug occurs:
/CUDA/benchmarks/t/wrap_error.cu
>40|    sumKernel<<<1, THREADS>>>(data_in, sum_out);
 41|    cudaDeviceSynchronize();
 42|
 43|    cudaFree(data_in);
 44|    cudaFree(sum_out);
 45|    return 0;

Note that in the above code line 13 is commented out, which disables the warp-level synchronization. 

13| //__syncwarp();

If line 13 is uncommented, the race will be fixed, and the tool will report no races: 

detected 0 races in total.

Detect races in a Makefile-based project

This tutorial shows how to use Coderrect to detect races in a Makefile-based project using pbzip2-0.9.4 as the example. pbzip2 is a parallel implementation of the bzip2 file compressor, and it contains a known race condition in version 0.9.4.

git clone https://github.com/sctbenchmarks/sctbenchmarks.git
cd sctbenchmarks/1CB-0-2/pbzip2-0.9.4/bzip2-1.0.6
make clean
make
cd ..
cd pbzip2-0.9.4
make clean
coderrect make

Prerequisites

This tutorial assumes you have successfully installed the Coderrect software following the quick start.

Background

In pbzip2, the program will spawn consumer threads that (de)compress the input file and spawn an output thread that writes data to the output file. However, the main thread only joins the output thread but does not join the consumer threads. So there is an order violation bug between the time when the main thread is destroying resources and the time when consumer threads are using the resources.

An interleaving that triggers the error looks like:

void main(...) { 
  ...
  for (i=0; i < numCPU; i++) {              
    ret = pthread_create(..., consumer, 
                              fifo);  
    ...             
  }
  ret = pthread_create(..., fileWriter, 
                            OutFilename);
  ...
  // start reading in data
  producer(..., fifo);
  ...
  // wait until exit of thread
  pthread_join(output, NULL);
  ...
  fifo->empty = 1;
  ...
  // reclaim memory
  queueDelete(fifo);
  fifo = NULL;



}

void *decompress_consumer(void *q) {
  ...
  for (;;) {


















    pthread_mutex_lock(fifo->mut);
    ...
  }
}

Since queueDelete will release the fifo queue used by consumer threads, the access on fifo->mut will result in a segmentation fault.

Detect the race using Coderrect

Make sure the code can be compiled

cd sctbenchmarks/1CB-0-2/pbzip2-0.9.4/pbzip2-0.9.4
make

You should see a pbzip2 executable under the same folder.

Run Coderrect

make clean 
coderrect -t -o report make

NOTE: The make clean command is to ensure there is no pre-built binaries so that Coderrect is able to analyze every piece of source code in the project.

The coderrect -t -o report make command will compile and analyze the problem, the reported races is stored under ./report directory as specified by -o option.

-t switch is used to generate a quick summary report in terminal.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in the browser.

Terminal Report

The terminal races reports looks like following:

==== Found a race between: 
line 1048, column 3 in pbzip2.cpp  AND  line 553, column 28 in pbzip2.cpp
Shared variable: 
 at line 991 of pbzip2.cpp
 991| q = new queue;
Thread 1: 
pbzip2.cpp@1048:3
Code snippet: 
 1046| pthread_mutex_destroy(q->mut);
 1047| delete q->mut;
>1048| q->mut = NULL;
 1049| }
 1050|
>>>Stack Trace:
>>>main
>>>  queueDelete(queue*) [pbzip2.cpp:1912]
Thread 2: 
pbzip2.cpp@553:28
Code snippet: 
 551| for (;;)
 552| {
>553| pthread_mutex_lock(fifo->mut);
 554| while (fifo->empty)
 555| {
>>>Stack Trace:
>>>pthread_create [pbzip2.cpp:1818]
>>>  consumer_decompress(void*) [pbzip2.cpp:1818]

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 1048, column 3 in pbzip2.cpp  AND  line 553, column 28 in pbzip2.cpp

Next the report shows the name and location of the variable on which the race occurs.

Shared variable: 
 at line 991 of pbzip2.cpp
 991| q = new queue;

This shows that the race occurs on the variable queue allocated at line 991.

Next the tool reports information about the two unsynchronized accesses to queue.
For each of the two accesses a location, code snippet, and stack trace is shown.

The location shows the file, line, and column of the access.

Thread 1: 
pbzip2.cpp@1048:3

So the above access occurs in pbzip2.cpp at line 1048 column 3.
The report also shows a preview of the file at that location.

Code snippet: 
 1046| pthread_mutex_destroy(q->mut);
 1047| delete q->mut;
>1048| q->mut = NULL;
 1049| }
 1050|

The code snippet shows that this access is a write to q->mut (set it to NULL).

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

Detect Fortran OpenMP Races

This tutorial shows how to use Coderrect to detect races in a single file multi-threaded Fortran program written with OpenMP.

Prerequisites

• This tutorial assumes you have successfully installed the Coderrect software following the quick start.

• Our sample code relies on OpenMP to achieve parallelism. You will need a compiler that supports OpenMP. We will be using gfortran, but other modern alternatives should also have OpenMP support. 

Detecting a race

We will be using a benchmark from DataRaceBench, a suite of OpenMP data race benchmarks designed to evaluate the effectiveness of data race detection tools developed by a group at Lawrence Livermore National Lab. DataRaceBench is full of great test cases (try using it to evaluate coderrect!).

We will be using the DRB001-antidep1-orig-yes.f95 case from DataRaceBench version 1.3.0.1. You can get the source code from the DataRaceBench repository on github, but we have included a snippet here for convenience.

!!!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~!!!
!!! Copyright (c) 2017-20, Lawrence Livermore National Security, LLC
!!! and DataRaceBench project contributors. See the DataRaceBench/COPYRIGHT file for details.
!!!
!!! SPDX-License-Identifier: (BSD-3-Clause)
!!!~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~!!!

!A loop with loop-carried anti-dependence.
!Data race pair: a[i+1]@25:9 vs. a[i]@25:16

program DRB001_antidep1_orig_yes
use omp_lib
    implicit none
    integer :: i, len
    integer :: a(1000)

    len = 1000

    do i = 1, len
        a(i) = i
    end do

    !$omp parallel do
    do i = 1, len-1
        a(i) = a(i+1) + 1
    end do
    !$omp end parallel do

    print 100, a(500)
    100 format ('a(500)=',i3)
end program

Start by checking that the program compiles successfully on your machine. Coderrect works by intercepting compiler calls. If the code cannot be compiled, Coderrect cannot run it’s analysis. The DRB001 benchmark can be built with the following command

gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001
./DRB001

You should see output that looks something like:

a(500)=502

Although your results may vary because there is a data race in this code. In the parallel for loop, the value of each iteration i depends on the next iteration i+1.

This means that thread 0 could be executing a[0] = a[1] + 1 at the same time thread 1 is running a[1] = a[2] + 1. Both threads are accessing a[1] in parallel, causing a data race.

Run Coderrect

The easiest way to run the tool is by passing the build command to coderrect:

coderrect -t gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001

Remember, the command to build DRB001 was gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001.

the -t switch is used to generate a quick summary report in terminal.

Calling coderrect in this way ensures all the required compilation flags can be passed on the command line. For a project using a build system such as make, coderrect tool can be called with the same build command used to build the project. For an example: checkout out detecting races in a Makefile-based project.

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing. The tool also generates a more comprehensive report that can be viewed in a browser.

Terminal Report

The terminal report for DRB001 should look something like the following:

==== Found a data race between: 
line 25, column 0 in DRB001-antidep1-orig-yes.f95 AND line 25, column 1 in DRB001-antidep1-orig-yes.f95
Shared variable:
 at line 0 of 
 0|
Thread 1:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do
>>>Stacktrace:
Thread 2:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do
>>>Stacktrace:

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 25, column 0 in DRB001-antidep1-orig-yes.f95 AND line 25, column 1 in DRB001-antidep1-orig-yes.f95

Next the report shows the variable name and location on which the race occurs. (Though this is sometimes not present for fortran programs)

Shared variable:
 ...

Going and finding the race location in the code may be a little tedious so the report also shows a preview of the file at that location.

Thread 1:
 23|    !$omp parallel do
 24|    do i = 1, len-1
>25|        a(i) = a(i+1) + 1
 26|    end do
 27|    !$omp end parallel do

The code snippet shows that this racing access is a write to a(i) as part of an OpenMP parallel for loop.

This is the race we expected to find for this DataRaceBench case. You can try running coderrect in the same way on the other DataRaceBench fortran benchmarks at dataracebench/micro-benchmarks-fortran.

HTML Report

The terminal is great to get a quick idea about what races are reported, but the full report can be viewed in a browser.

We can also save the race report to a directory specified via the command option-o <dir>. 

coderrect -o report gfortran -fopenmp DRB001-antidep1-orig-yes.f95 -o DRB001

This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Exclude Uninteresting Code Paths

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

By default, Coderrect traverses every function that is reachable from the main function (or from any of the user-specified entry functions). However, in many cases, users might regard certain functions as uninteresting in their own projects and therefore do not want Coderrect to analyze those functions. This can be done by setting configuration for “skipFunctions” in .coderrect.json:

//.coderrect.json
"skipFunctions":[
        "print_msg"
] 

In the above example, the function “print_msg” will not be analyzed by Coderrect.

Analyzing Interesting Indirect Calls

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

To make Coderrect fast, in the default setting Coderrect only analyzes a limited number of indirect function calls (functions that are invoked through function pointers) for each call site. It is because that the number of targets that can be pointed by the function pointer can be large and analyzing each of them is time-consuming. This works well for common cases in practice. However, in cases where a call site has many targets, users can configure “indirectFunctions” to analyze all possible targets called in a function. For example:

"indirectFunctions":[
        "OpBase_Consume"
] 

In the above, OpBase_Consume is a function from RedisGraph and it contains an indirect call op->consume(op) with a large number of possible call targets:

inline Record OpBase_Consume(OpBase *op) {
        return op->consume(op);
}

The configuration instructs Coderrect to analyze all possible indirect call targets in the function “OpBase_Consume“.

Define Custom Lock APIs

Quick Start

To make Coderrect understand your custom lock APIs, please open your configuration file (the default one is $installDir/conf/coderrect.json), and add the following lines:

{
        ... ...
        lockUnlockFunctions: {
            "MyMutexLockFunc": "MyMutexUnlockFunc",
            "MySpinLockFunc": "MySpinUnlockFunc"
        }
}

Introduction

Lock APIs (e.g. pthread_mutex) are the most common synchronization mechanism used by developers. Coderrect has the built-in support for std::lock, std::mutex, pthread_mutex, and pthread_spin_lock. However, developers may use third-party libraries that have their own lock APIs. Furthermore, developers may create their own lock APIs to fit the special needs. 

Coderrect relies on the configuration “lockUnlockFunctions” to recognize these proprietary APIs.

lockUnlockFunctions

The code below creates a naive custom spin lock and protects simultaneous access to the global variable “x” by using it. 

#include <iostream>
#include <atomic>
#include <thread>

struct my_spinlock_t {
    std::atomic<bool> a;

    my_spinlock_t() : a(false) {}
};

void MySpinLock(my_spinlock_t* lock) {
    bool expected = false;
    while (!lock->a.compare_exchange_strong(expected, true))
    ;
}

void MySpinUnlock(my_spinlock_t* lock) {
    lock->a = false;
}

my_spinlock_t lock;
int x;


void thread_function() {
    MySpinLock(&lock);
    x++;
    std::cout << x << "n";
    MySpinUnlock(&lock);
}


int main ()
{
    std::thread first (thread_function);
    std::thread second (thread_function);
    first.join();
    second.join();
    return 0;
}

Let’s run Coderrect with the default configuration:

Coderrect detects a race because it doesn’t understand the meaning of MySpinLock and MySpinUnlcok and determines “x” are accessed by two threads simultaneously without synchronization.

Now, let’s tell Coderrect the meaning of MySpinLock/MySpinUnlock with a custom local configuration file lockunlockfunc.json:

{
    "lockUnlockFunctions": {
         "MySpinLock": "MySpinUnlock"
    }
}

The lockUnlockFunctions is a list of key-value pairs. Each pair has the name of the lock function as the key and the name of the unlock function as the value.

Let’s apply this configuration file to Coderrect and try again

You tell Coderrect to load and overlap the configuration file “lockunlockfunc.json” to the default coderrect.json using “-conf” option. Now, Coderrect knows MySpinLock and MySpinUnlock are a pair of lock-unlock functions and makes correct reasoning of the thread interleave.

Support Version

Coderrect 0.7.0+

Common Configurations

Detect races in an open source project Memcached

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Coderrect detected 3 new races (2 of them were confirmed) in memcached. To detect the reported bugs using coderrect, download and checkout the buggy version of memcached using the following commands.

$ git clone https://github.com/memcached/memcached.git
$ cd memcached
$ git checkout 82029ecc9b3dd0f57b3f9ab9761f44714cceed6f

Detect the race

1. Build memcached using coderrect

# install dependencies
$ apt install libevent-dev
# configure memcached
$ ./autogen.sh && ./configure
# build memcached using coderrect
$ coderrect -t make

The coderrect -t make command will compile and analyze the program automatically. 

2. Detect races using coderrect

After compilation, coderrect automatically detects and lists all the potential targets to analyzed as follows: 

1) timedrun
2) sizes
3) memcached-debug
4) memcached
5) testapp
Please select binaries by entering their ordinal numbers (e.g. 1,2,6):

Select 3) memcached-debug as the target to detect races on the debug version of memcached.

Interpret the Results

The coderrect tool generates a comprehensive report that can be viewed in a browser. 

HTML Report

To view the full report, open ‘.coderrect/report/index.html‘ in your browser.

The HTML report looks like the following picture.

Terminal Report

To get a quick overview of the detected races, coderrect can also report a summary of the most interesting races in the terminal (with -t flag, checkout all coderrect options). The terminal races report looks like the following:

==== Found a race between: 
line 162, column 5 in crawler.c AND line 1464, column 16 in items.c
Shared variable:
 at line 1577 of items.c
 1577|        calloc(1, sizeof(struct crawler_expired_data));
Thread 1:
 160|    pthread_mutex_lock(&d->lock);
 161|    d->crawlerstats[slab_cls].end_time = current_time;
>162|    d->crawlerstats[slab_cls].run_complete = true;
 163|    pthread_mutex_unlock(&d->lock);
 164|}
>>>Stacktrace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    lru_crawler_class_done [crawler.c:378]
>>>      crawler_expired_doneclass [crawler.c:350]
Thread 2:
 1462|        crawlerstats_t *s = &cdata->crawlerstats[i];
 1463|        /* We've not successfully kicked off a crawl yet. */
>1464|        if (s->run_complete) {
 1465|            char *lru_name = "na";
 1466|            pthread_mutex_lock(&cdata->lock);
>>>Stacktrace:
>>>pthread_create [items.c:1703]
>>>  lru_maintainer_thread [items.c:1703]
>>>    lru_maintainer_crawler_check [items.c:1647]

Each reported race starts with a summary of where the race was found.

==== Found a race between: 
line 162, column 5 in crawler.c AND line 1464, column 16 in items.c

Next the report shows the name and location of the variable on which the race occurs.

Shared variable:
 at line 1577 of items.c
 1577|        calloc(1, sizeof(struct crawler_expired_data));

For example, the above result shows that the race occurs on the variable allocated at line 1577 in items.c file.

Next the tool reports information about the two unsynchronized accesses to the shared variable. For each of the two accesses, the code snippet, and the stack trace is shown.

Since finding the reported location in the code may be a little tedious, the report shows a preview of the file at that location.

Thread 1:
 160|    pthread_mutex_lock(&d->lock);
 161|    d->crawlerstats[slab_cls].end_time = current_time;
>162|    d->crawlerstats[slab_cls].run_complete = true;
 163|    pthread_mutex_unlock(&d->lock);
 164|}

The code snippet shows that the race is on variable crawlerstats[slab_cls].run_complete. Coderrect also shows the stack trace that triggers the race to make validation on the race simpler.

>>>Stacktrace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    lru_crawler_class_done [crawler.c:378]
>>>      crawler_expired_doneclass [crawler.c:350]

Detecting Races in Redis

Redis is a popular in-memory database that persists on disk. Redis has been heavily adopted and is widely used. The GitHub repository has over 43,000 stars, over 18,000 questions have been tagged with [redis] on StackOverflow, and there are various meetups and conferences for Redis every year.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Detect the race

Redis is easy to build. To detect races on redis, simply run the following:

git clone -b 6.0.0 https://github.com/antirez/redis.git
cd redis
coderrect -t -o report -e redis-server,redis-benchmark make -j

• First, we clone the buggy release (6.0.0). 
• Next, we cd into the redis directory and run the Coderrect tool to detect races. 
• The -t flag will print the race report to the terminal so that we can quickly investigate some of the races. 
• The -o report flag tells the tool to generate a report and place it into a directory named “report”. 
• Next, Redis builds multiple executables by default (redis-server, redis-benchmark, and redis-cli). Coderrect is capable of analyzing one or more targets at once. Here we specify that Coderrect should analyze both redis-server and red-benchmark with the flag  -e redis-server,redis-benchmark. 
• Lastly, we add the command make -j to build redis.

The analysis time for each project will be printed to the terminal as is shown below:

Analyzing /path/to/redis/src/redis-benchmark ...
 - ✔ [00m:01s] Loading IR From File                    
 - ✔ [00m:02s] Running Compiler Optimization Passes (Phase I)                                              
 - ✔ [00m:00s] Canonicalizing Loops                    
 - ✔ [00m:00s] Propagating Constants                     
 - ✔ [00m:00s] Running Compiler Optimization Passes (Phase II)                                               
 - ✔ [00m:00s] Running Pointer Analysis                        
 - ✔ [00m:09s] Building Static Happens-Before Graph                                    
 - ✔ [00m:11s] Detecting Races               
 - ✔ [00m:00s] Scanning for additional OpenMP Regions                                                                        

Analyzing /path/to/redis/src/redis-server ...
 - ✔ [00m:04s] Loading IR From File                    
...


----------------------------The summary of races in redis-server------------------------

 12 shared data races

----------------------------The summary of races in redis-benchmark------------------------

  3 shared data races

Coderrect provide an option to list all potential target binaries and specify which binaries should be analyzed with the following command.

coderrect make

Coderrect will automatically detect which binaries were built using make and list the potential targets to be analyzed as shown below.

The project creates multiple executables. Please select one from the list below
to detect races. 

In the future, you can specify the executable using the option "-e" if you know 
which ones you want to analyze. 

    coderrect -e your_executable_name1,your_executable_name2 your_build_command_line

 1) src/redis-benchmark
 2) src/redis-cli
 3) src/redis-server
 4) deps/lua/src/luac
 5) deps/lua/src/lua

Please select binaries by entering their ordinal numbers (e.g. 1,2,6):1,2,3

In addition, Coderrect allow users to analyze all generated binaries by using -analyzeAllBinaries flag to command line. Coderrect will skip the asking stage and automatically analyze all binaries.

coderrect -analyzeAllBinaries make

Interpret the Results

The coderrect tool reports a quick summary of the most interesting races directly in the terminal for quick viewing when run with -t. The tool also generates a more comprehensive report that can be viewed in the browser. 

Terminal Report

Next we will explain an example of a race reported in redis-server, as well as how to debug the race based on the provided information:

==== Found a race between: 
line 888, column 9 in debug.c AND line 885, column 16 in debug.c
Shared variable:
 at line 72 of server.c
 72|struct redisServer server; /* Server global state */
Thread 1:
 886|        serverLogRaw(LL_WARNING|LL_RAW,
 887|        "nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n");
>888|        server.bug_report_start = 1;
 889|    }
 890|}
>>>Stacktrace:
>>>pthread_create [bio.c:123]
>>>  bioProcessBackgroundJobs [bio.c:123]
>>>    lazyfreeFreeObjectFromBioThread [bio.c:209]
>>>      decrRefCount [lazyfree.c:130]
>>>        freeListObject [object.c:365]
>>>          _serverPanic [object.c:291]
>>>            bugReportStart [debug.c:873]
Thread 2:
 883|
 884|void bugReportStart(void) {
>885|    if (server.bug_report_start == 0) {
 886|        serverLogRaw(LL_WARNING|LL_RAW,
 887|        "nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n");
>>>Stacktrace:
>>>pthread_create [networking.c:2942]
>>>  IOThreadMain [networking.c:2942]
>>>    _serverAssert [networking.c:2891]
>>>      bugReportStart [debug.c:791]

At a glance we can quickly see that the race is between the lines 885 and 888 in the debug.c file. Line 888 writes to server.bug_report_start and line 885 reads server.bug_report_start. To confirm this race we need to show 1) both accesses can be made in parallel, and 2) the parallel accesses are being made to the same server.bug_report_start object.

884|void bugReportStart(void) {
885|  if (server.bug_report_start == 0) {
886|    serverLogRaw(LL_WARNING|LL_RAW,
887|    "nn=== REDIS BUG REPORT START: ...n");
888|    server.bug_report_start = 1;
889|  }
890|}

To prove that 1) both accesses can be made in parallel, we need to show that thread 1 and 2 can both call the bugReportStart function at the same time. We can check the stack trace to see if there are any synchronizations between the threads after they are spawned.

Thread 1
>>>Stacktrace:
>>>pthread_create [bio.c:123]
>>>  bioProcessBackgroundJobs [bio.c:123]
>>>    lazyfreeFreeObjectFromBioThread [bio.c:209]
>>>      decrRefCount [lazyfree.c:130]
>>>        freeListObject [object.c:365]
>>>          _serverPanic [object.c:291]
>>>            bugReportStart [debug.c:873]
Thread 2:
>>>Stacktrace:
>>>pthread_create [networking.c:2942]
>>>  IOThreadMain [networking.c:2942]
>>>    _serverAssert [networking.c:2891]
>>>      bugReportStart [debug.c:791]

The first stack trace shows that thread 1 is spawned at bio.c:123 inside of the bioInit function. The second stack trace shows that thread 2 is spawned at networking.c:2942 in the initThreadedIO function. By looking one level up the stack trace, we can see that both bioInit and initThreadedIO are called one after the other from InitServerLast in server.c. 

void InitServerLast() {
    bioInit();
    initThreadedIO();

If we check the body of the bioInit function we see there are no synchronizations or joins after thread 1 is spawned. This means thread 1 will continue to execute even after the main thread has returned from bioInit and entered initThreadIO. Then, while in initThreadIO the main thread spawns thread 2 in parallel with thread 1. Thus we have shown that both threads do indeed run in parallel.

Next we must show that 2) the parallel accesses are being made to the same server.bug_report_start object. The report also shows us the creation site of the shared object on which the race occurs (in this case server).

Shared variable:
 at line 72 of server.c

By inspecting server.c:72 we see that server is actually a global variable! This means that that both threads are indeed accessing the same object in memory.

72| struct redisServer server; /* Server global state */

So, by showing that 1) both accesses can be made in parallel and 2) the parallel accesses are being made to the same server.bug_report_startobject we have confirmed this race in redis.

HTML Report

The terminal is great to get a quick idea about what races are reported, but a full and more detailed report can be viewed in the browser.

When we initially ran coderrect we included the -o report flag. This created a directory named report and a file named index.html within that directory. To view the full report open the index.html file in a browser.

Depending on your system this can be done through the terminal with some command. E.g. browse report/index.html on Gnome based systems or open report/index.html on Mac.

Detecting Races in RedisGraph

RedisGraph is a popular graph database module of Redis that supports using linear algebra to query the graph. This tutorial shows how to run coderrect on RedisGraph for finding potential races step by step.

1. Download RedisGraph and setup dependency:

$ git clone --recurse-submodules https://github.com/RedisGraph/RedisGraph.git
$ cd RedisGraph
$ apt-get install build-essential cmake m4 automake peg libtool autoconf

2. Run coderrect:

$ coderrect -t make

Running coderrect -t make builds the RedisGraph module redisgraph.so, which is a dynamic library. To analyze the library, Coderrect prompts a list of APIs for users to select as entry points: 

Analyzing /RedisGraph/src/redisgraph.so ...
   1) AC_GetU32  
   2) AGPLN_AddAfter    
   3) AGPLN_AddBefore                
   ...
   122) CommandDispatch
   123) ConcurrentCmdCtx_KeepRedisCtx                             
   124) ConcurrentSearch_HandleRedisCommand

Please select APIs by entering their numbers or names:

3. Select an entry point for analysis:

Enter “122” or type “CommandDispatch“. Coderrect then continues to analyze the library with “CommandDispatch” as an entry point:

 - â–– [00m:24s] Loading IR From File          
EntryPoints:
CommandDispatch
 - ✔ [00m:24s] Loading IR From File                    
 - ✔ [00m:40s] Running Compiler Optimization Passes (Phase I)                                              
 - ✔ [00m:00s] Canonicalizing Loops                    
 - ✔ [00m:00s] Propagating Constants                     
 - ✔ [03m:09s] Running Compiler Optimization Passes (Phase II)                                               
 - ✔ [00m:00s] Running Pointer Analysis                        
 - ✔ [03m:32s] Building Static Happens-Before Graph                                    
 - ✔ [00m:10s] Detecting Races               
 - ✔ [03m:39s] Scanning for additional OpenMP Regions

4. Analysis results:

In a few minutes, Coderrect generates the following results:

----------------------The summary of races in redisgraph.so------------------

==== Found a race between: 
line 389, column 21 in graph/graphcontext.c AND line 385, column 31 in graph/graphcontext.c
Shared variable:
 at line 39 of module.c
 39|GraphContext **graphs_in_keyspace;  // Global array tracking all extant GraphContexts.
Thread 1:
 387| if(graphs_in_keyspace[i] == gc) return;
 388| }
>389| graphs_in_keyspace = array_append(graphs_in_keyspace, gc);
 390|}
 391|
>>>Stacktrace:
>>>CommandDispatch
>>>  GraphContext_Retrieve [commands/cmd_dispatcher.c:68]
>>>    _GraphContext_Create [graph/graphcontext.c:132]
>>>      GraphContext_RegisterWithModule [graph/graphcontext.c:103]
Thread 2:
 383|void GraphContext_RegisterWithModule(GraphContext *gc) {
 384| // See if the graph context is not already in the keyspace.
>385| uint graph_count = array_len(graphs_in_keyspace);
 386| for(uint i = 0; i < graph_count; i ++) {
 387| if(graphs_in_keyspace[i] == gc) return;
>>>Stacktrace:
>>>CommandDispatch
>>>  GraphContext_Retrieve [commands/cmd_dispatcher.c:68]
>>>    _GraphContext_Create [graph/graphcontext.c:132]
>>>      GraphContext_RegisterWithModule [graph/graphcontext.c:103]...

  7 shared data races
 22 OpenMP races

To check the race report, please open '/git/RedisGraph/.coderrect/report/index.html' in your browser

Coderrect detected 7 shared data races and 22 OpenMP races in total in RedisGraph. The full report can be viewed in the browser.

Analyze static/dynamic library code

How to detect races in library code without a “main” method? This tutorial showcases this feature with a simple RingBuffer library libringbuffer.a, available on github.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Checkout the code and run coderrect with the following commands:

$ git clone https://github.com/coderrect/tutorial.git
$ cd tutorial/ringbuffer && cmake .
$ coderrect -t make

Coderrect will detect two public APIs in this library

1) RingBuffer::Consume       2) RingBuffer::Publish    

Please select APIs by entering their numbers or names (e.g. 1,2,RingBuffer::Consume,RingBuffer::Publish): 

In the terminal, type 1,2,3 and press Enter, Coderrect will report a race:

==== Found a race between: 
line 24, column 13 in ringbuffer_lib.cpp AND line 31, column 13 in ringbuffer_lib.cpp
Thread 1: 
 22|            buffer_[write_pos_] = value;
 23|            write_pos_++;
>24|            available_++;
 25|            return true;
 26|        }
>>>Stack Trace:
>>>pthread_create
>>>  coderrect_cb.1
>>>    RingBuffer::Publish(int)
Thread 2: 
 29|    }
 30|    bool RingBuffer::Consume(int *r) {
>31|        if (available_ == 0) {
 32|            return false;
 33|        }
>>>Stack Trace:
>>>pthread_create
>>>  coderrect_cb.2
>>>    RingBuffer::Consume(int*)
detected 1 races in total.

Behind the scene, Coderrect “simulates” the library’s behavior that RingBuffer::Publish and RingBuffer::Consume can be executed simultaneously by multiple threads. Even though there exists no “main” function, Coderrect will create two concurrent threads to invoke each API and detects the race.

Configuring Entry Points 

The library contains two public APIs: “RingBuffer::Publish” and “RingBuffer::Consume”, which can be called by multiple threads on a shared RingBuffer. To detect races in this library, users can also specify these APIs as entry points in a configuration file “.coderrect.json“

// .coderrect.json
{
  "entryPoints": [
    "RingBuffer::Publish",
    "RingBuffer::Consume"
  ]
}

Note: the namespace “RingBuffer::” can be skipped when no ambiguity exists.

Place the above configuration file under where you run Coderrect. Then run coderrect -t make again. You will see that the same race is reported.

More Advanced Option

Even better, configuring the entry points is unnecessary, but these entry points can be discovered automatically by Coderrect with the option -racedetect.analyzeApi. 

$ coderrect -t -racedetect.analyzeApi make

You will see that the same race as before is reported without any configuration. 

Caveat: Coderrect currently does not infer if an API must be executed before another. If that’s the case, false positives may be generated. To avoid them, explicitly configure entry points. 

The full library code is shown below.

//"ringbuffer_lib.h"
#ifndef RINGBUFFER_LIB_H
#define RINGBUFFER_LIB_H

#include <cstddef>
#include <stdexcept>
    class RingBuffer {
    private:
        int *buffer_;
        size_t write_pos_;
        size_t available_;
        size_t capacity_;

    public:
        RingBuffer(size_t capacity);
        ~RingBuffer();

        bool Publish(int value);
        bool Consume(int *r);
    };

#endif //RINGBUFFER_LIB_H

//"ringbuffer_lib.cpp"
  
#include "ringbuffer_lib.h"

    RingBuffer::RingBuffer(size_t capacity) : capacity_(capacity) {
        if (capacity == 0)
            throw std::invalid_argument("capacity must be greater than 0");

        buffer_ = new int[capacity];
        available_ = 0;
        write_pos_ = 0;
    }
    RingBuffer::~RingBuffer() {
        if (buffer_ != nullptr)
            delete[] buffer_;
    }
    bool RingBuffer::Publish(int value) {
        if (available_ < capacity_) {
            if (write_pos_ >= capacity_) {
                write_pos_ = 0;
            }
            buffer_[write_pos_] = value;
            write_pos_++;
            available_++;
            return true;
        }

        return false;
    }
    bool RingBuffer::Consume(int *r) {
        if (available_ == 0) {
            return false;
        }
        int next_slot = write_pos_ - available_;
        if (next_slot < 0) {
            next_slot += capacity_;
        }
        *r = buffer_[next_slot];
        available_--;
        return true;
    }

Here is the file “CMakeLists.txt“:

cmake_minimum_required(VERSION 3.13)
project(ringbuffer)
set(CMAKE_CXX_STANDARD 11)
find_package(Threads)
add_library(ringbuffer STATIC
            ringbuffer_lib.cpp
            ringbuffer_lib.h)

Coderrect Fast and Exhaust Modes

At different phases of the software development, developers may have different time and resource constraints. When editing code in an IDE, users may like to run Coderrect repeatedly and see the results as fast as possible. When testing code before a release, users may want to run Coderrect overnight on machines with large memory and core counts to exhaustively check all possible errors. 

Pay-as-you-go detection

To fit in different use scenarios, Coderrect offers pay-as-you-go detection: the more time you afford to run Coderrect, the better results, i.e., more code coverage and more precise results, you will likely get.

Coderrect has three modes: default, fast, and exhaust. These modes can be configured by the -mode option:

$ coderrect -mode=[default|fast|exhaust] 

Default mode aims to fit most uses. It exploits a balance between performance, code coverage and precision. The default mode is pretty fast and scalable: it can analyze over a million lines of C/C++ code such as the Linux kernel in less than ten minutes. 

Fast mode is optimized for speed. For large code base it can be many times faster than the default mode, even though it may possibly lose some analysis precision. At code development time, users may prefer to run Coderrect with “-mode=fast” to turn on the fast mode to reduce the waiting time. 

$ coderrect -mode=fast make

Exhaust mode is optimized for coverage and will exhaustively check all possible races under all code paths, all files and modules, as well as all call chains and dependencies. 

At nightly build time, you may prefer to run Coderrect with “-mode=exhaust” to find all possible race conditions as precise as possible when there is enough time to run (e.g., eight hours): 

$ coderrect -mode=exhaust make

Sample Performance Results on Redis

The following shows the performance of the fast mode on analyzing the Redis server:

The fast mode finishes in less than two minutes and detects 12 races:

The following shows the performance of the exhaust mode on analyzing the Redis server:

The exhaust mode finishes in around 75 mins and detects 22 races:

Blacklist and rank race conditions through configuration

How to config Coderrect to intentionally ignore certain race conditions or rank them by priority? This tutorial showcases this feature on an open-source project: memcached.

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

Run the following commands to clone the Git repository of memcached, install the libevent dependency, configure the build, and then run Coderrect:

$ git clone https://github.com/memcached/memcached.git
$ cd memcached
$ apt-get install libevent-dev autotools-dev automake
$ ./autogen.sh
$ ./configure
$ coderrect -t make

Detecting races

In a few seconds, you are expected to see a list of executables built by memcached:

The project creates multiple executables. Please select one from the list below
to detect races. 
In the future, you can specify the executable using the option "-e" if you know 
which ones you want to analyze. 
    coderrect -e your_executable_name1,your_executable_name2 your_build_command_line
 1) timedrun
 2) sizes
 3) memcached-debug
 4) memcached
 5) testapp

Select 3 to detect races in memcached-debug. Coderrect will report:

line 128, column 9 in assoc.c AND line 107, column 28 in assoc.c
Shared variable: 
hashpower at line 35 of assoc.c
 35|unsigned int hashpower = HASHPOWER_DEFAULT;
Thread 1: 
 126|        if (settings.verbose > 1)
 127|            fprintf(stderr, "Hash table expansion startingn");
>128|        hashpower++;
 129|        expanding = true;
 130|        expand_bucket = 0;
>>>Stack Trace:
>>>pthread_create [assoc.c:277]
>>>  assoc_maintenance_thread [assoc.c:277]
>>>    assoc_expand [assoc.c:256]
Thread 2: 
 105|
 106|    if (expanding &&
>107|        (oldbucket = (hv & hashmask(hashpower - 1))) >= expand_bucket)
 108|    {
 109|        pos = &old_hashtable[oldbucket];
>>>Stack Trace:
>>>pthread_create [crawler.c:505]
>>>  item_crawler_thread [crawler.c:505]
>>>    crawler_expired_eval [crawler.c:419]
>>>      do_item_unlink_nolock [crawler.c:219]
>>>        assoc_delete [items.c:531]
>>>          _hashitem_before [assoc.c:172]
...

...
...
...
detected 25 races in total.

So Coderrect detected 23 shared data races, 2 mismatched API issues, 4 TOCTOU. However, the races reported might be false positives.

Blacklisting races

To ignore race detections in certain part of the code that you don’t care about, you can configure Coderrect in three ways: 

1) ignore the function entirely through option “ignoreRacesInFunctions”,

2) ignore certain locations by file name and line number through option “ignoreRacesAtLocations”, and

3) ignore certain variables by their name through option “ignoreRaceVariables”.

All you need to do is to write a config file named coderrect.json

// .coderrect.json
{
  "ignoreRacesInFunctions": [
        "assoc_expand",
        "logger_create"
    ],
  "ignoreRacesAtLocations": [
        "items.c:1277",
        "extstore.c:493"
    ],
  "ignoreRaceVariables": [
        "crawlers",
        "wrap"
    ]
}

and put it under the path where you run Coderrect (In this case, it is the root path of memcached).

The config file above allows Coderrect to bypass all methods whose name matches “assoc_expand” or “logger_create”, ignore potential races in file “items.c” line “1277” and file “extstore.c” line “493“, and on global variables “crawlers” and “wrap“.

Now rerun Coderrect

$ make clean && coderrect -t make

Only 19 shared data races, 2 mismatched API issues, 3 TOCTOU were reported this time (and all of them are likely real true races):

Configure race priority

If you think certain races are more critical or less critical than others, you can configure them through the options “lowPriorityRaces“, “lowPriorityFiles“, and “highPriorityRaces“:

"highPriorityRaces": [
        "heads",
        "tails"
    ],
"lowPriorityRaces": [
        "stats*",
        "settings*"
    ]

The config above allows Coderrect to rank races whose variable’s names match “heads” and “tails” higher in the report, and those match “stats*”and “settings*” lower in the report. 

"lowPriorityFiles": [
        "*debug*"
    ],
"highPriorityFiles": [
        "*mem*"
    ]

The config above allows Coderrect to rank races whose file names match “*mem*” higher in the report, and those match “*debug*” lower in the report.

Can They Run in Parallel? A Must-Not-Run-In-Parallel Analysis

This tutorial assumes that you have gone through one of the three starter case tutorials and have successfully run Coderrect.

While locks and semaphores are frequently used to prevent data races in critical sections, developers also use standard synchronizations such as condition variables and barriers to prevent threads from running in parallel on a pair of functions. Coderrect performs a pretty precise “must-not-run-in-parallel” analysis that takes into consideration most common synchronization APIs such as pthread fork, join, wait, signal, broadcast, barrier, etc, as well as mutually exclusive branch conditions. More technical details can be found in a research paper.

If the user’s code relies on customized synchronizations to prevent threads from running in parallel, such as the once-only execution in the Open vSwitch OVS project. Coderrect may not recognize them and hence may report false warnings.

To address the issue, Coderrect supports user-specified not-run-in-parallel APIs through the configuration file .coderrect.json.

For example, adding the following to .coderrect.json will help Coderrect to recognize function1 and function2 cannot run in parallel. Thus Coderrect will not report data races between these two functions (as well as code called from them).

//.coderrect.json"
"notParallelFunctionPairs": {
    "function1" : "function2"
}

You can add multiple pairs of such functions and can also use wildcard (*) to match with any function. For example, bioInit and bioProcessBackgroundJobs are one pair of not-run-in-parallel functions,  ovsrcu_init_module and ovsrcu_init_module are another pair, meaning that ovsrcu_init_module can only be executed once and it cannot race with itself.

taosThreadToOpenNewFile and * are another pair, meaning that taosThreadToOpenNewFile can only be executed in a sequential environment and thus cannot race with any other function.

"notParallelFunctionPairs": {
    "bioInit":"bioProcessBackgroundJobs",
    "ovsrcu_init_module":"ovsrcu_init_module",
    "taosThreadToOpenNewFile" : "*"
}

Note that for each pair declared in notParallelFunctionPairs, the order of the specified APIs are interchangeable.

Coderrect and Jenkins Integration (Using Jenkinsfile)

This tutorial shows how to integrate Coderrect with Jenkins, the most popular CI/CD platform.

Coderrect is a static analysis tool for C/C++ code. Unlike popular tools such as Cppcheck and Clang static analyzer, one of the powerful features of Coderrect is the detection of concurrency races caused by code using explicit multi-thread APIs (e.g. PThread, std::thread) or implicit parallel APIs (e.g. OpenMP). 

Besides running Coderrect to scan the local source code via the command line, you can easily integrate it with Jenkins to include it in your CI/CD pipeline so that Coderrect can be triggered automatically by a pull request, a code check-in, or a scheduled nightly build.

Overview

The diagram below illustrates the interaction among components in Coderrect-Jenkins integration.

When a build is scheduled Jenkins master asks Jenkins agents to prepare the source code (usually pull the latest code from a source control system), build the project, test and analysis the program. 

Upon receiving the request from Jenkins Master, the Jenkins agent runs Coderrect installed locally to analyze the code and report discoveries back to the master. 

Install Coderrect

The first step is to install the Coderrect package to the Jenkins agent machine. Note that Coderrect just supports Linux for now. 

1. Download the latest version of Coderrect from here (the source can be found here).
2. Unpack the downloaded tarball.
3. Add /path/to/coderrect-installation/bin into PATH.

Install Coderrect Jenkins Plugin

You can download Coderrect Jenkins plugin here. Then go to the “Advanced” tab of “Manage Plugins” page.

For “Upload Plugin”, click “Choose File” to select the downloaded coderrect.hpi. 

Create a multi-branch project

Now we can create a multi-branch project in Jenkins.

Go to Jenkins portal and click “New Item“.

Enter the project name and select “Multibranch Pipeline” 

Set up the source control and build information. Make sure you use Jenkinsfile to control the build pipeline.

Create Jenkinsfile

We already created a project in Jenkins. To let Jenkins know how to build, test and analyze the code, we have to create a file called Jenkinsfile under the root directory of your repository.

pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh "coderrect make"
            }
        }

        stage('Analysis') {
            steps {
                publishCoderrect ""
            }
        }
    }
}

The Jenkinsfile above defines a pipeline with two stages. The “build” stage builds the project and runs coderrect to analyze the generated binary. The “analysis” stage reports the analysis results to the master.

You may be aware that the command “publishCoderrect” has a parameter whose value is “” in our example. What is it? It is the relative path (to the project root directory) where you executes “coderrect”. We execute “coderrect” immediately under the project root so that we specify its value to “”.

If we execute “coderrect” under a directory rather than the project root we need to set up this parameter correctly. For example, the following figure shows typical build steps for a cmake-based project:

# we are under the project root directory
mkdir build
cd build
cmake ..
coderrect make

In this case you execute “coderrect” under project-root/build. You need to set the build directory to “build”.

#pipeline {
    agent any

    stages {
        stage('Build') {
            steps {
                sh "mkdir build && cd build && cmake .. && coderrect make"
            }
        }

        stage('Analysis') {
            steps {
                     publishCoderrect "build"
            }
        }
    }
}

Now we are done! We can go to the project dashboard and manually trigger a build.

Trigger a Build

Coderrect adds a section into the build summary page. The section shows the number of races in different categories for this build and the previous build. For example, the demo project above has 13 data races.

You can click the “Coderrect” item in the navigation bar to see details.

The detail report has three views – full, new and solved. The “full” view shows all races found from the current build. The “new” view shows just races introduced by the new committed code since previous build. The “solved” view shows races fixed by the new code.

I hope you enjoy this post. Please send to feedback@coderrect.com if you have any feedback.

Coderrect and Jenkins Integration (Using freestyle project)

This tutorial shows how to integrate Coderrect with Jenkins, the most popular CI/CD platform.

Coderrect is a static analysis tool for C/C++ code. Unlike popular tools such as Cppcheck and Clang static analyzer, one of the powerful features of Coderrect is the detection of concurrency races caused by code using explicit multi-thread APIs (e.g. PThread, std::thread) or implicit parallel APIs (e.g. OpenMP). 

Besides running Coderrect to scan the local source code via the command line, you can easily integrate it with Jenkins to include it in your CI/CD pipeline so that Coderrect can be triggered automatically by a pull request, a code check-in, or a scheduled nightly build.

Overview

The diagram below illustrates the interaction among components in Coderrect-Jenkins integration.

When a build is scheduled Jenkins master asks Jenkins agents to prepare the source code (usually pull the latest code from a source control system), build the project, test and analysis the program. 

Upon receiving the request from Jenkins Master, the Jenkins agent runs Coderrect installed locally to analyze the code and report discoveries back to the master. 

Install Coderrect

The first step is to install the Coderrect package to the Jenkins agent machine. Note that Coderrect just supports Linux for now. 

1. Download the latest version of Coderrect from here (the source can be found here).
2. Unpack the downloaded tarball.
3. Add /path/to/coderrect-installation/bin into PATH.

Install Coderrect Jenkins Plugin

You can download Coderrect Jenkins plugin file here. Then go to the “Advanced” tab of “Manage Plugins” page.

For “Upload Plugin”, click “Choose File” to select the downloaded coderrect.hpi. 

Create a Freestyle project

Now we can create a freestyle project in Jenkins.

Go to Jenkins portal and click “New Item“.

Enter the project name (TDEngine here) and select “Freestyle project” 

Set up the source control and build information. For the build stage, I create the following script commands

The last step “coderrect -e taosd -t make” tells coderrect to

• Execute “make” to build binaries
• Analyze an executable called “taosd” to detect races
• Generate a terminal-based race report besides the default HTML-based report

Set up the post-build stage by choosing “Publish the race report to Jenkins”

Since we run “coderrect” under the project_root/build, we specify a relative path “build” so that coderrect knows where to find the race report.

You are ready to go ffter you click “Save”!

Trigger a Build

Coderrect adds a section into the build summary page. The section shows the number of races in different categories for this build and the previous build. For example, the demo project above has 13 data races.

You can click the “Coderrect” item in the navigation bar to see detail report.

The detail report has three views – full, new and solved. The “full” view shows all races found from the current build. The “new” view shows just races introduced by the new committed code since previous build. The “solved” view shows races fixed by the new code.

I hope you enjoy this post. Please send to feedback@coderrect.com if you have any feedback.

Github CI/CD Integration

If you are already running a CI/CD service on Github, it is easy to add Coderrect to your workflows. In a nutshell, add the following to .github/workflows/ci.yml

- name: coderrect setup
        run: |
            wget https://public-installer-pkg.s3.us-east-2.amazonaws.com/coderrect-linux-1.1.3.tar.gz
            tar zxf coderrect-linux-1.1.3.tar.gz
            echo "$PWD/coderrect-linux-1.1.3/bin" >> $GITHUB_PATH
- name: make
       run: coderrect -analyzeAllBinaries make
- name: coderrect check
        run: coderrect -publishResults

Background

Continuous Integration/Continuous Delivery (CI/CD) is a common practice now for developing software. In CI, typically tests and checks are run against every pull request and code commit to ensure the new code changes do not break anything or introduce new bugs. CI makes it easier to fix bugs quickly and often.

If your code is hosted on Github, you can create custom CI/CD workflows directly in your GitHub repository with GitHub Actions. All you need is to add a .yml file under .github/workflows/ of your Github repo.

As an example, Redis has a ci.yml file with the following content:

name: CI
on: [push, pull_request]
jobs:
    test-ubuntu-latest:
        runs-on: ubuntu-latest
        steps:
          - uses: actions/checkout@v2
          - name: make
            run: make
          - name: test
            run: |
                sudo apt-get install tcl8.5
               ./runtest --verbose
         - name: module api test
            run: ./runtest-moduleapi --verbose

With the configuration above, Github CI will build and test Redis upon every code push and pull request. If the build is successful and the tests are all passed, Github will show a green tick associated with the commit. 

Otherwise, if the build or any test fails, you will see a red cross: 

Integrating Coderrect into Github CI

To run coderrect in the Github CI, in ci.yml, simply add the following three steps:

# Step 1: download coderrect and add to path
- name: setup coderrect
        run: |
           wget https://public-installer-pkg.s3.us-east-2.amazonaws.com/coderrect-linux-0.8.0.tar.gz
           tar zxf coderrect-linux-0.8.0.tar.gz
           echo "::add-path::$PWD/coderrect-linux-0.8.0/bin"

# Step 2: build and run coderrect
- name: make
        run: coderrect -analyzeAllBinaries make

# Step 3: coderrect results
- name: coderrect check
        run: coderrect -publishResults

In Step 1, coderrect is setup; 

In Step 2, the flag -analyzeAllBinaries tells Coderrect to check all the build targets;

In Step 3, if any race is detected the results will be reported in the CI terminal. A red cross sign will be shown, indicating potential race hazards found by Coderrect: 

To see the detailed race information, just click the arrow sign to expand it: 

Cache the downloaded Coderrect

Coderrect only needs to be downloaded once for Github CI. To cache the downloaded Coderrect package, simply add the following to ci.yml.

- name: cache coderrect
    id: cache-coderrect
    uses: actions/cache@v2
    with:
        path: coderrect-linux-0.8.0
        key: ${{ runner.os }}-coderrect

- name: setup coderrect
    if: steps.cache-coderrect.outputs.cache-hit != 'true'

A full example is available at here.

Installing Coderrect with Spack

Spack is an increasingly popular package manager for Linux and macOS that eliminates the difficulties of managing complex software dependencies. Spack is especially popular in the high performance computing community. Spack won the prestigious R&D 100 Award in 2019 and is the official deployment tool of the ExaScale Computing Project which is tasked finding the tools and systems to support the next generation of ExaScale super computers in the U.S. Spack can even be used to set up a cloud-HPC cluster in AWS.

Prerequisites

This tutorial assumes you already have Spack installed. If you are new to Spack, we recommend following Spack’s Getting Started Guide.

Installing Coderrect with Spack

First download and add the Coderrect Spack repository to your local machine.

$ wget https://github.com/coderrect-inc/coderrect-spack/archive/master.zip
$ unzip master.zip && rm master.zip
$ mv coderrect-spack-master coderrect-spack
$ spack repo add coderrect-spack

Once the repo is registered, Spack makes its easy to install and load the Coderrect tool.

$ spack install coderrect
$ spack load coderrect

Coderrect is now installed and ready to use. Spack really does make things simple!

Check out the Coderrect Quick Start page or some of the other tutorials to see examples of how to use the Coderrect tool.

If you are already using GitHub Action as your CI/CD service, it is easy to integrate Coderrect to your workflows.

In a nutshell, add the following to your .github/workflows/ci.yml:

- name: Coderrect Scan
  uses: coderrect-inc/coderrect-github-action@main 

Background

Continuous Integration/Continuous Delivery (CI/CD) is a common practice now for developing software. In CI, typically tests and checks are run against every pull request and code commit to ensure the new code changes do not break anything or introduce new bugs. CI makes it easier to fix bugs quickly and often.

If your code is hosted on GitHub, you can create custom CI/CD workflows directly in your GitHub repository with GitHub Actions.

In this tutorial, we take memcached as an example project to demonstrate how to set up GitHub Actions.

To start, you can click the “Action“ tab above your GitHub Repository.

Once you enter the Action tab, GitHub will guide you to create your own CI/CD script by providing different template scripts based on different building system.

Here we select “set up a workflow yourself“ to use the basic template.

GitHub will automatically create a YAML file (main.yml by default) under .github/workflows and this is the path which you should put your future scripts under. The left-hand side is the script editor and the right-hand side is the place you can search different existing Action scripts published in GitHub Marketplace.

The default template script looks similar as below, we will explain it in detail:

# This is a basic workflow to help you get started with Actions
name: CI

# Controls when the action will run.
on:
  push:
    branches: [ master ]
  pull_request:
    branches: [ master ]

  # Allows you to run this workflow manually from the Actions tab
   workflow_dispatch:

# Defines the workflow of this action.
jobs:
  # This workflow contains a single job called "build"
  build:
    # The type of runner that the job will run on
    runs-on: ubuntu-latest 

    # Steps represent a sequence of tasks that will be executed as part of the job
     steps:
       # Checks-out your repository under $GITHUB_WORKSPACE, so your job can access it
       - uses: actions/checkout@v2

    # Runs a single command using the runners shell
    - name: Run a one-line script
      run: echo Hello, world!

    # Runs a set of commands using the runners shell
    - name: Run a multi-line script
      run: |
        echo Add other actions to build,
        echo test, and deploy your project.

The definition of an action consists of two major parts: on and jobs.

Field on defines when this action will be triggered. The default template only triggers the action when there’s a push or pull request made upon the main branch.

For example, if you want the action to be triggered at any event on any branch, you can simply change the script to:

on: [push, pull_request]

If you only want to trigger actions manually, then you should specify workflow_dispatch instead. This will create a button in the Action Tab for you to trigger them manually later. Check out this blog on how to manually trigger GitHub Actions.

Field job defines the actual workflow of the action. The subfield build is a customized job name, and you can change it to something more meaningful, such as test-build-on-ubuntu.

runs-on specifies the system image on which you want to run your CI/CD tasks.

steps specifies the detailed sub-tasks performed in a job. You can compose your CI/CD job using the published Actions from GitHub Marketplace and your own customized scripts.

Integrating Coderrect into Github CI

Coderrect provides a GitHub Action in Marketplace for you to integrate it into your CI/CD process easily.

In general, integrating Coderrect requires 3 steps:

steps: 
# step 1 
- uses: actions/checkout@v2 
# step 2 
- name: Install deps
  run: |
    sudo apt-get update -y
    sudo apt-get install -y libevent-dev libseccomp-dev git libsasl2-dev 
- name: Build
  run: |
    gcc --version
    ./autogen.sh
    ./configure --enable-seccomp --enable-tls --enable-sasl --enable-sasl-pwdb
    make -j 
# step 3 
- name: coderrect scan
  uses: coderrect-inc/coderrect-github-action@main

Step 1 checks out your GitHub repository, using an action provided by GitHub.

Step 2 installs all dependencies required for build the project (“Install deps”) and does a test build (“Build“). Note that including task “Build“ is not required for Coderrect to function, but it’s critical to make sure your project can successfully build before applying Coderrect.

Step 3 applies Coderrect to your project. You can search the GitHub Marketplace to obtain the most updated script and all available options.

Once this script is saved, the GitHub Action you just defined will be automatically triggered when you push new commits or merge pull requests. You can review them by entering the “Action“ tab.

By clicking the specific task (“Update ci.yml“ in the example above), you can view its detailed results.

To see Coderrect’s race detection report, you can click task “coderrect scan“ to expand its terminal output. Coderrect will output a list of summary for all executables it analyzes and also attach an report link at the end for you to view them in detail.

Configure Coderrect GitHub Action:

If you are using a different building system (e.g., CMake) or a different language or compiler, then you need to properly configure Coderrect. Here we provide instructions for some commonly used configurations, more detailed instructions can be found on our GitHub Action page and Coderrect documentation.

For CMake projects

You will need to install and setup cmake first.

- name: download cmake
  run: |
    wget https://cmake.org/files/v3.18/cmake-3.18.2-Linux-x86_64.tar.gz
    tar xf cmake-3.18.2-Linux-x86_64.tar.gz
    mkdir build && cd build
    ../cmake-3.18.2-Linux-x86_64/bin/cmake .. 

One significant difference for CMake projects is that you need to specify the build path. For example, if you are building your project under ./build, then you need to specify the build path using following script:

- name: Coderrect Scan
  uses: coderrect-inc/coderrect-github-action@main
  with:
    buildPath: "build"

For FORTRAN projects

You will need to install the FORTRAN compiler. For example:

- name: Install fortran
  run: |
    sudo apt-get update -y
    sudo apt-get install -y gfortran

Then it is likely that you need to specify the FORTRAN compiler when you use make.

If so, Coderrect provides an optional input called buildCommand to specify your customized build command.

- name: coderrect scan
  uses: coderrect-inc/coderrect-github-action@v1.2
  with:
    buildCommand: "make COMPILER=GNU MPI_COMPILER=gfortran C_MPI_COMPILER=gcc"

For customized analysis

Coderrect’s GitHub Action provides an input called options that allows you to provide arbitrary options supported in Coderrect’s CLI. A common use case is to provide Coderrect with a fully-customized JSON configuration using -conf.

In order to do so, check our documentation to see available configuration items. Once you write a configuration file (say coderrect.json). You can pass it to the scanner as below:

- name: coderrect scan
  uses: coderrect-inc/coderrect-github-action@v1.2
  with:
    options: "-analyzeAllBinaries -conf=/path/to/coderrect.json"

The path should be a relative path from your build directory (e.g., if your build directory is ./build/ and your config file is under the root path, then you should specify the config file as “-conf=../coderrect.json”).