{"id":4331,"date":"2020-12-13T14:43:06","date_gmt":"2020-12-13T20:43:06","guid":{"rendered":"https:\/\/coderrect.com\/?post_type=docs&p=4331"},"modified":"2021-01-06T15:14:00","modified_gmt":"2021-01-06T21:14:00","slug":"typical-cases","status":"publish","type":"docs","link":"https:\/\/coderrect.com\/docs\/typical-cases\/","title":{"rendered":"Typical Cases"},"content":{"rendered":"\n
<\/p>\n\n\n\n
This tutorial assumes that you have gone through one of the three starter case tutorials<\/a> and have successfully run Coderrect.<\/p>\n\n\n\n Coderrect detected\u00a03 new races<\/a>\u00a0(2 of them were confirmed) in\u00a0memcached<\/a>. To detect the reported bugs using coderrect, download and checkout the buggy version of memcached using the following commands.<\/p>\n\n\n\n The\u00a0coderrect -t make<\/em><\/strong>\u00a0command will compile and analyze the program automatically.\u00a0<\/p>\n\n\n\n After compilation, coderrect automatically detects and lists all the potential targets to analyzed as follows:\u00a0<\/p>\n\n\n\n Select 3)\u00a0memcached-debug<\/em><\/strong>\u00a0as the target to detect races on the debug version of memcached.<\/p>\n\n\n\n The\u00a0coderrect<\/em><\/strong>\u00a0tool generates a comprehensive report that can be viewed in a browser.\u00a0<\/p>\n\n\n\n To view the full report, open \u2018.coderrect\/report\/index.htm<\/em><\/strong>l\u2018 in your browser.<\/p>\n\n\n\n The HTML report looks like the following picture.<\/p>\n\n\n\n To get a quick overview of the detected races, coderrect can also report a summary of the most interesting races in the terminal (with\u00a0-t<\/em><\/strong>\u00a0flag, checkout\u00a0all coderrect options<\/a>). The terminal races report looks like the following:<\/p>\n\n\n\n Each reported race starts with a summary of where the race was found.<\/p>\n\n\n\n Next the report shows the name and location of the variable on which the race occurs.<\/p>\n\n\n\n For example, the above result shows that the race occurs on the variable\u00a0allocated at line 1577 in\u00a0items.c<\/em><\/strong>\u00a0file.<\/p>\n\n\n\n Next the tool reports information about the two unsynchronized accesses to the shared variable. For each of the two accesses, the code snippet, and the stack trace is shown.<\/p>\n\n\n\n Since finding the reported location in the code may be a little tedious, the report shows a preview of the file at that location.<\/p>\n\n\n\n The code snippet shows that the race is on variable\u00a0crawlerstats[slab_cls].run_complete<\/em><\/strong>. Coderrect also shows the stack trace that triggers the race to make validation on the race simpler.<\/p>\n\n\n\n Redis<\/a>\u00a0is a popular in-memory database that persists on disk. Redis has been heavily adopted and is widely used. The\u00a0GitHub repository<\/a>\u00a0has over 43,000 stars, over 18,000 questions have been tagged with\u00a0[redis]<\/em><\/strong>\u00a0on\u00a0StackOverflow<\/a>, and there are various meetups and conferences for Redis\u00a0every year<\/a>.<\/p>\n\n\n\n This tutorial assumes that you have gone through one of the three starter case tutorials<\/a> and have successfully run Coderrect.<\/p>\n\n\n\n Redis is easy to build. To detect races on redis, simply run the following:<\/p>\n\n\n\n The analysis time for each project will be printed to the terminal as is shown below:<\/p>\n\n\n\n Coderrect provide an option to list all potential target binaries and specify which binaries should be analyzed with the following command.<\/p>\n\n\n\n Coderrect will automatically detect which binaries were built using make and list the potential targets to be analyzed as shown below.<\/p>\n\n\n\n In addition, Coderrect allow users to analyze\u00a0all<\/strong>\u00a0generated binaries by using\u00a0-analyzeAllBinaries<\/strong><\/em>\u00a0flag to command line. Coderrect will skip the asking stage and automatically analyze all binaries.<\/p>\n\n\n\n The\u00a0coderrect\u00a0<\/em><\/strong>tool reports a quick summary of the most interesting races directly in the terminal for quick viewing when run with\u00a0-t<\/strong><\/em>. The tool also generates a more comprehensive report that can be viewed in the\u00a0browser<\/em><\/strong>.\u00a0<\/p>\n\n\n\n Next we will explain an example of a race reported in redis-server, as well as how to debug the race based on the provided information:<\/p>\n\n\n\n At a glance we can quickly see that the race is between the lines\u00a0885<\/a>\u00a0and\u00a0888<\/a>\u00a0in the debug.c file. Line 888 writes to\u00a0server.bug_report_start<\/em><\/strong>\u00a0and line 885 reads\u00a0server.bug_report_start<\/em><\/strong>. To confirm this race we need to show 1) both accesses can be made in parallel, and 2) the parallel accesses are being made to the same\u00a0server.bug_report_start<\/em><\/strong>\u00a0object.<\/p>\n\n\n\n To prove that 1) both accesses can be made in parallel, we need to show that thread 1 and 2 can both call the\u00a0bugReportStart<\/a>\u00a0function at the same time. We can check the stack trace to see if there are any synchronizations between the threads after they are spawned.<\/p>\n\n\n\n The first stack trace shows that thread 1 is spawned at\u00a0bio.c:123<\/a>\u00a0inside of the\u00a0bioInit<\/a>\u00a0function. The second stack trace shows that thread 2 is spawned at\u00a0networking.c:2942<\/a>\u00a0in the\u00a0initThreadedIO<\/a>\u00a0function. By looking one level up the stack trace, we can see that both\u00a0bioInit<\/a>\u00a0and\u00a0initThreadedIO<\/a>\u00a0are called one after the other from\u00a0InitServerLast<\/a>\u00a0in server.c.\u00a0<\/p>\n\n\n\n If we check the body of the\u00a0bioInit<\/a>\u00a0function we see there are no synchronizations or joins after thread 1 is spawned. This means thread 1 will continue to execute even after the main thread has returned from\u00a0bioInit<\/a>\u00a0and entered\u00a0initThreadIO<\/a>. Then, while in\u00a0initThreadIO<\/a>\u00a0the main thread spawns thread 2 in parallel with thread 1. Thus we have shown that both threads do indeed run in parallel.<\/p>\n\n\n\n Next we must show that 2) the parallel accesses are being made to the same\u00a0server.bug_report_start<\/em><\/strong>\u00a0object. The report also shows us the creation site of the shared object on which the race occurs (in this case\u00a0server<\/a>).<\/p>\n\n\n\n By inspecting\u00a0server.c:72<\/a>\u00a0we see that\u00a0server<\/a>\u00a0is actually a global variable! This means that that both threads are indeed accessing the same object in memory.<\/p>\n\n\n\n So, by showing that 1) both accesses can be made in parallel and 2) the parallel accesses are being made to the same\u00a0server.bug_report_startobject<\/em><\/strong> we have confirmed this race in redis.<\/p>\n\n\n\n The terminal is great to get a quick idea about what races are reported, but a full and more detailed report can be viewed in the browser.<\/p>\n\n\n\n When we initially ran coderrect we included the\u00a0-o report<\/em><\/strong>\u00a0flag. This created a directory named\u00a0report<\/em><\/strong>\u00a0and a file named\u00a0index.html<\/em><\/strong>\u00a0within that directory. To view the full report open the\u00a0index.html<\/em><\/strong>\u00a0file in a browser.<\/p>\n\n\n\n$ git clone https:\/\/github.com\/memcached\/memcached.git\n$ cd memcached\n$ git checkout 82029ecc9b3dd0f57b3f9ab9761f44714cceed6f<\/pre>\n\n\n\n
\n\n\n\nDetect the race<\/h3>\n\n\n\n
# install dependencies\n$ apt install libevent-dev\n# configure memcached\n$ .\/autogen.sh && .\/configure\n# build memcached using coderrect\n$ coderrect -t make<\/pre>\n\n\n\n
1) timedrun\n2) sizes\n3) memcached-debug\n4) memcached\n5) testapp\nPlease select binaries by entering their ordinal numbers (e.g. 1,2,6):<\/pre>\n\n\n\n
\n\n\n\nInterpret the Results<\/h3>\n\n\n\n
HTML Report<\/h4>\n\n\n\n
<\/figure>\n\n\n\n
Terminal Report<\/h4>\n\n\n\n
==== Found a race between: \nline 162, column 5 in crawler.c AND line 1464, column 16 in items.c\nShared variable:\n at line 1577 of items.c\n 1577| calloc(1, sizeof(struct crawler_expired_data));\nThread 1:\n 160| pthread_mutex_lock(&d->lock);\n 161| d->crawlerstats[slab_cls].end_time = current_time;\n>162| d->crawlerstats[slab_cls].run_complete = true;\n 163| pthread_mutex_unlock(&d->lock);\n 164|}\n>>>Stacktrace:\n>>>pthread_create [crawler.c:505]\n>>> item_crawler_thread [crawler.c:505]\n>>> lru_crawler_class_done [crawler.c:378]\n>>> crawler_expired_doneclass [crawler.c:350]\nThread 2:\n 1462| crawlerstats_t *s = &cdata->crawlerstats[i];\n 1463| \/* We've not successfully kicked off a crawl yet. *\/\n>1464| if (s->run_complete) {\n 1465| char *lru_name = \"na\";\n 1466| pthread_mutex_lock(&cdata->lock);\n>>>Stacktrace:\n>>>pthread_create [items.c:1703]\n>>> lru_maintainer_thread [items.c:1703]\n>>> lru_maintainer_crawler_check [items.c:1647]<\/pre>\n\n\n\n
==== Found a race between: \nline 162, column 5 in crawler.c AND line 1464, column 16 in items.c<\/pre>\n\n\n\n
Shared variable:\n at line 1577 of items.c\n 1577| calloc(1, sizeof(struct crawler_expired_data));<\/pre>\n\n\n\n
Thread 1:\n 160| pthread_mutex_lock(&d->lock);\n 161| d->crawlerstats[slab_cls].end_time = current_time;\n>162| d->crawlerstats[slab_cls].run_complete = true;\n 163| pthread_mutex_unlock(&d->lock);\n 164|}<\/pre>\n\n\n\n
>>>Stacktrace:\n>>>pthread_create [crawler.c:505]\n>>> item_crawler_thread [crawler.c:505]\n>>> lru_crawler_class_done [crawler.c:378]\n>>> crawler_expired_doneclass [crawler.c:350]<\/pre>\n\n\n\n
\n\n\n\nDetecting Races in Redis<\/h2>\n\n\n\n
\n\n\n\nDetect the race<\/h3>\n\n\n\n
git clone -b 6.0.0 https:\/\/github.com\/antirez\/redis.git\ncd redis\ncoderrect -t -o report -e redis-server,redis-benchmark make -j<\/pre>\n\n\n\n
Analyzing \/path\/to\/redis\/src\/redis-benchmark ...\n - \u00e2\u0153\u201d [00m:01s] Loading IR From File \n - \u00e2\u0153\u201d [00m:02s] Running Compiler Optimization Passes (Phase I) \n - \u00e2\u0153\u201d [00m:00s] Canonicalizing Loops \n - \u00e2\u0153\u201d [00m:00s] Propagating Constants \n - \u00e2\u0153\u201d [00m:00s] Running Compiler Optimization Passes (Phase II) \n - \u00e2\u0153\u201d [00m:00s] Running Pointer Analysis \n - \u00e2\u0153\u201d [00m:09s] Building Static Happens-Before Graph \n - \u00e2\u0153\u201d [00m:11s] Detecting Races \n - \u00e2\u0153\u201d [00m:00s] Scanning for additional OpenMP Regions \n\nAnalyzing \/path\/to\/redis\/src\/redis-server ...\n - \u00e2\u0153\u201d [00m:04s] Loading IR From File \n...\n\n\n----------------------------The summary of races in redis-server------------------------\n\n 12 shared data races\n\n----------------------------The summary of races in redis-benchmark------------------------\n\n 3 shared data races<\/pre>\n\n\n\n
\n\n\n\ncoderrect make<\/pre>\n\n\n\n
The project creates multiple executables. Please select one from the list below\nto detect races. \n\nIn the future, you can specify the executable using the option \"-e\" if you know \nwhich ones you want to analyze. \n\n coderrect -e your_executable_name1,your_executable_name2 your_build_command_line\n\n 1) src\/redis-benchmark\n 2) src\/redis-cli\n 3) src\/redis-server\n 4) deps\/lua\/src\/luac\n 5) deps\/lua\/src\/lua\n\nPlease select binaries by entering their ordinal numbers (e.g. 1,2,6):1,2,3<\/pre>\n\n\n\n
coderrect -analyzeAllBinaries make<\/pre>\n\n\n\n
\n\n\n\nInterpret the Results<\/h3>\n\n\n\n
\n\n\n\nTerminal Report<\/h4>\n\n\n\n
==== Found a race between: \nline 888, column 9 in debug.c AND line 885, column 16 in debug.c\nShared variable:\n at line 72 of server.c\n 72|struct redisServer server; \/* Server global state *\/\nThread 1:\n 886| serverLogRaw(LL_WARNING|LL_RAW,\n 887| \"nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n\");\n>888| server.bug_report_start = 1;\n 889| }\n 890|}\n>>>Stacktrace:\n>>>pthread_create [bio.c:123]\n>>> bioProcessBackgroundJobs [bio.c:123]\n>>> lazyfreeFreeObjectFromBioThread [bio.c:209]\n>>> decrRefCount [lazyfree.c:130]\n>>> freeListObject [object.c:365]\n>>> _serverPanic [object.c:291]\n>>> bugReportStart [debug.c:873]\nThread 2:\n 883|\n 884|void bugReportStart(void) {\n>885| if (server.bug_report_start == 0) {\n 886| serverLogRaw(LL_WARNING|LL_RAW,\n 887| \"nn=== REDIS BUG REPORT START: Cut & paste starting from here ===n\");\n>>>Stacktrace:\n>>>pthread_create [networking.c:2942]\n>>> IOThreadMain [networking.c:2942]\n>>> _serverAssert [networking.c:2891]\n>>> bugReportStart [debug.c:791]\n<\/pre>\n\n\n\n
884|void bugReportStart(void) {\n885| if (server.bug_report_start == 0) {\n886| serverLogRaw(LL_WARNING|LL_RAW,\n887| \"nn=== REDIS BUG REPORT START: ...n\");\n888| server.bug_report_start = 1;\n889| }\n890|}<\/pre>\n\n\n\n
Thread 1\n>>>Stacktrace:\n>>>pthread_create [bio.c:123]\n>>> bioProcessBackgroundJobs [bio.c:123]\n>>> lazyfreeFreeObjectFromBioThread [bio.c:209]\n>>> decrRefCount [lazyfree.c:130]\n>>> freeListObject [object.c:365]\n>>> _serverPanic [object.c:291]\n>>> bugReportStart [debug.c:873]\nThread 2:\n>>>Stacktrace:\n>>>pthread_create [networking.c:2942]\n>>> IOThreadMain [networking.c:2942]\n>>> _serverAssert [networking.c:2891]\n>>> bugReportStart [debug.c:791]<\/pre>\n\n\n\n
void InitServerLast() {\n bioInit();\n initThreadedIO();<\/pre>\n\n\n\n
Shared variable:\n at line 72 of server.c<\/pre>\n\n\n\n
72| struct redisServer server; \/* Server global state *\/<\/pre>\n\n\n\n
\n\n\n\nHTML Report<\/h4>\n\n\n\n
<\/figure>\n\n\n\n