{ "schema_version": 2, "id": "operate/oss_and_stack/management/debugging", "title": "Debugging", "url": "https://redis.io/docs/latest/operate/oss_and_stack/management/debugging/", "summary": "A guide to debugging Redis server processes", "tags": [ "docs", "operate", "stack", "oss" ], "last_updated": "2024-01-30T17:50:06+01:00", "page_type": "content", "content_hash": "100df3edd9fcf7e0835de9c6f71f694a050c1c8752e2f63e0f4c37efbe029a05", "sections": [ { "id": "overview", "title": "Overview", "role": "overview", "text": "Redis is developed with an emphasis on stability. We do our best with\nevery release to make sure you'll experience a stable product with no\ncrashes. However, if you ever need to debug the Redis process itself, read on.\n\nWhen Redis crashes, it produces a detailed report of what happened. However,\nsometimes looking at the crash report is not enough, nor is it possible for\nthe Redis core team to reproduce the issue independently. In this scenario, we\nneed help from the user who can reproduce the issue.\n\nThis guide shows how to use GDB to provide the information the\nRedis developers will need to track the bug more easily." }, { "id": "what-is-gdb", "title": "What is GDB?", "role": "content", "text": "GDB is the Gnu Debugger: a program that is able to inspect the internal state\nof another program. Usually tracking and fixing a bug is an exercise in\ngathering more information about the state of the program at the moment the\nbug happens, so GDB is an extremely useful tool.\n\nGDB can be used in two ways:\n\n* It can attach to a running program and inspect the state of it at runtime.\n* It can inspect the state of a program that already terminated using what is called a *core file*, that is, the image of the memory at the time the program was running.\n\nFrom the point of view of investigating Redis bugs we need to use both of these\nGDB modes. The user able to reproduce the bug attaches GDB to their running Redis\ninstance, and when the crash happens, they create the `core` file that in turn\nthe developer will use to inspect the Redis internals at the time of the crash.\n\nThis way the developer can perform all the inspections in his or her computer\nwithout the help of the user, and the user is free to restart Redis in their\nproduction environment." }, { "id": "compiling-redis-without-optimizations", "title": "Compiling Redis without optimizations", "role": "content", "text": "By default Redis is compiled with the `-O2` switch, this means that compiler\noptimizations are enabled. This makes the Redis executable faster, but at the\nsame time it makes Redis (like any other program) harder to inspect using GDB.\n\nIt is better to attach GDB to Redis compiled without optimizations using the\n`make noopt` command (instead of just using the plain `make` command). However,\nif you have an already running Redis in production there is no need to recompile\nand restart it if this is going to create problems on your side. GDB still works\nagainst executables compiled with optimizations.\n\nYou should not be overly concerned at the loss of performance from compiling Redis\nwithout optimizations. It is unlikely that this will cause problems in your\nenvironment as Redis is not very CPU-bound." }, { "id": "attaching-gdb-to-a-running-process", "title": "Attaching GDB to a running process", "role": "content", "text": "If you have an already running Redis server, you can attach GDB to it, so that\nif Redis crashes it will be possible to both inspect the internals and generate\na `core dump` file.\n\nAfter you attach GDB to the Redis process it will continue running as usual without\nany loss of performance, so this is not a dangerous procedure.\n\nIn order to attach GDB the first thing you need is the *process ID* of the running\nRedis instance (the *pid* of the process). You can easily obtain it using\n`redis-cli`:\n\n $ redis-cli info | grep process_id\n process_id:58414\n\nIn the above example the process ID is **58414**.\n\nLogin into your Redis server.\n\n(Optional but recommended) Start **screen** or **tmux** or any other program that will make sure that your GDB session will not be closed if your ssh connection times out. You can learn more about screen in [this article](http://www.linuxjournal.com/article/6340).\n\nAttach GDB to the running Redis server by typing:\n\n $ gdb \n\nFor example:\n\n $ gdb /usr/local/bin/redis-server 58414\n\nGDB will start and will attach to the running server printing something like the following:\n\n Reading symbols for shared libraries + done\n 0x00007fff8d4797e6 in epoll_wait ()\n (gdb)\n\nAt this point GDB is attached but **your Redis instance is blocked by GDB**. In\norder to let the Redis instance continue the execution just type **continue** at\nthe GDB prompt, and press enter.\n\n (gdb) continue\n Continuing.\n\nDone! Now your Redis instance has GDB attached. Now you can wait for the next crash. :)\n\nNow it's time to detach your screen/tmux session, if you are running GDB using it, by\npressing **Ctrl-a a** key combination." }, { "id": "after-the-crash", "title": "After the crash", "role": "content", "text": "Redis has a command to simulate a segmentation fault (in other words a bad crash) using\nthe `DEBUG SEGFAULT` command (don't use it against a real production instance of course!\nSo I'll use this command to crash my instance to show what happens in the GDB side:\n\n (gdb) continue\n Continuing.\n\n Program received signal EXC_BAD_ACCESS, Could not access memory.\n Reason: KERN_INVALID_ADDRESS at address: 0xffffffffffffffff\n debugCommand (c=0x7ffc32005000) at debug.c:220\n 220 *((char*)-1) = 'x';\n\nAs you can see GDB detected that Redis crashed, and was even able to show me\nthe file name and line number causing the crash. This is already much better\nthan the Redis crash report back trace (containing just function names and\nbinary offsets)." }, { "id": "obtaining-the-stack-trace", "title": "Obtaining the stack trace", "role": "content", "text": "The first thing to do is to obtain a full stack trace with GDB. This is as\nsimple as using the **bt** command:\n\n (gdb) bt\n #0 debugCommand (c=0x7ffc32005000) at debug.c:220\n #1 0x000000010d246d63 in call (c=0x7ffc32005000) at redis.c:1163\n #2 0x000000010d247290 in processCommand (c=0x7ffc32005000) at redis.c:1305\n #3 0x000000010d251660 in processInputBuffer (c=0x7ffc32005000) at networking.c:959\n #4 0x000000010d251872 in readQueryFromClient (el=0x0, fd=5, privdata=0x7fff76f1c0b0, mask=220924512) at networking.c:1021\n #5 0x000000010d243523 in aeProcessEvents (eventLoop=0x7fff6ce408d0, flags=220829559) at ae.c:352\n #6 0x000000010d24373b in aeMain (eventLoop=0x10d429ef0) at ae.c:397\n #7 0x000000010d2494ff in main (argc=1, argv=0x10d2b2900) at redis.c:2046\n\nThis shows the backtrace, but we also want to dump the processor registers using the **info registers** command:\n\n (gdb) info registers\n rax 0x0 0\n rbx 0x7ffc32005000 140721147367424\n rcx 0x10d2b0a60 4515891808\n rdx 0x7fff76f1c0b0 140735188943024\n rsi 0x10d299777 4515796855\n rdi 0x0 0\n rbp 0x7fff6ce40730 0x7fff6ce40730\n rsp 0x7fff6ce40650 0x7fff6ce40650\n r8 0x4f26b3f7 1327936503\n r9 0x7fff6ce40718 140735020271384\n r10 0x81 129\n r11 0x10d430398 4517462936\n r12 0x4b7c04f8babc0 1327936503000000\n r13 0x10d3350a0 4516434080\n r14 0x10d42d9f0 4517452272\n r15 0x10d430398 4517462936\n rip 0x10d26cfd4 0x10d26cfd4 \n eflags 0x10246 66118\n cs 0x2b 43\n ss 0x0 0\n ds 0x0 0\n es 0x0 0\n fs 0x0 0\n gs 0x0 0\n\nPlease **make sure to include** both of these outputs in your bug report." }, { "id": "obtaining-the-core-file", "title": "Obtaining the core file", "role": "content", "text": "The next step is to generate the core dump, that is the image of the memory of the running Redis process. This is done using the `gcore` command:\n\n (gdb) gcore\n Saved corefile core.58414\n\nNow you have the core dump to send to the Redis developer, but **it is important\nto understand** that this happens to contain all the data that was inside the\nRedis instance at the time of the crash; Redis developers will make sure not to\nshare the content with anyone else, and will delete the file as soon as it is no\nlonger used for debugging purposes, but you are warned that by sending the core\nfile you are sending your data." }, { "id": "what-to-send-to-developers", "title": "What to send to developers", "role": "content", "text": "Finally you can send everything to the Redis core team:\n\n* The Redis executable you are using.\n* The stack trace produced by the **bt** command, and the registers dump.\n* The core file you generated with gdb.\n* Information about the operating system and GCC version, and Redis version you are using." }, { "id": "thank-you", "title": "Thank you", "role": "content", "text": "Your help is extremely important! Many issues can only be tracked this way. So\nthanks!" } ], "examples": [] }