{
  "schema_version": 2,
  "id": "operate/oss_and_stack/management/optimization/latency",
  "title": "Diagnosing latency issues",
  "url": "https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency/",
  "summary": "Finding the causes of slow responses",
  "aliases": [
    "/manual/optimization/latency/",
    "/management/optimization/latency/"
  ],
  "tags": [
    "docs",
    "operate",
    "stack",
    "oss"
  ],
  "last_updated": "2026-09-21T15:59:03+01:00",
  "page_type": "content",
  "content_hash": "e1139aec0cac462d485087f2f52226cfcd9e7a3947f62fc4f19c5ac561ba0a6c",
  "sections": [
    {
      "id": "content",
      "title": "Content",
      "role": "content",
      "text": "This document will help you understand what the problem could be if you\nare experiencing latency problems with Redis.\n\nIn this context *latency* is the maximum delay between the time a client\nissues a command and the time the reply to the command is received by the\nclient. Usually Redis processing time is extremely low, in the sub microsecond\nrange, but there are certain conditions leading to higher latency figures.\n\nI've little time, give me the checklist\n---\n\nThe following documentation is very important in order to run Redis in\na low latency fashion. However I understand that we are busy people, so\nlet's start with a quick checklist. If you fail following these steps, please\nreturn here to read the full documentation.\n\n1. Make sure you are not running slow commands that are blocking the server. Use the Redis [Slow Log feature](https://redis.io/docs/latest/commands/slowlog) to check this.\n2. For EC2 users, make sure you use HVM based modern EC2 instances, like m3.medium. Otherwise fork() is too slow.\n3. Transparent huge pages must be disabled from your kernel. Use `echo never > /sys/kernel/mm/transparent_hugepage/enabled` to disable them, and restart your Redis process.\n4. If you are using a virtual machine, it is possible that you have an intrinsic latency that has nothing to do with Redis. Check the minimum latency you can expect from your runtime environment using `./redis-cli --intrinsic-latency 100`. Note: you need to run this command in *the server* not in the client.\n5. Enable and use the [Latency monitor](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency-monitor) feature of Redis in order to get a human readable description of the latency events and causes in your Redis instance.\n\nIn general, use the following table for durability VS latency/performance tradeoffs, ordered from stronger safety to better latency.\n\n1. AOF + fsync always: this is very slow, you should use it only if you know what you are doing.\n2. AOF + fsync every second: this is a good compromise.\n3. AOF + fsync every second + no-appendfsync-on-rewrite option set to yes: this is as the above, but avoids to fsync during rewrites to lower the disk pressure.\n4. AOF + fsync never. Fsyncing is up to the kernel in this setup, even less disk pressure and risk of latency spikes.\n5. RDB. Here you have a vast spectrum of tradeoffs depending on the save triggers you configure.\n\nAnd now for people with 15 minutes to spend, the details...\n\nMeasuring latency\n-----------------\n\nIf you are experiencing latency problems, you probably know how to measure\nit in the context of your application, or maybe your latency problem is very\nevident even macroscopically. However redis-cli can be used to measure the\nlatency of a Redis server in milliseconds, just try:\n\n    redis-cli --latency -h `host` -p `port`\n\nUsing the internal Redis latency monitoring subsystem\n---\n\nSince Redis 2.8.13, Redis provides latency monitoring capabilities that\nare able to sample different execution paths to understand where the\nserver is blocking. This makes debugging of the problems illustrated in\nthis documentation much simpler, so we suggest enabling latency monitoring\nASAP. Please refer to the [Latency monitor documentation](https://redis.io/docs/latest/operate/oss_and_stack/management/optimization/latency-monitor).\n\nWhile the latency monitoring sampling and reporting capabilities will make\nit simpler to understand the source of latency in your Redis system, it is still\nadvised that you read this documentation extensively to better understand\nthe topic of Redis and latency spikes.\n\nLatency baseline\n----------------\n\nThere is a kind of latency that is inherently part of the environment where\nyou run Redis, that is the latency provided by your operating system kernel\nand, if you are using virtualization, by the hypervisor you are using.\n\nWhile this latency can't be removed it is important to study it because\nit is the baseline, or in other words, you won't be able to achieve a Redis\nlatency that is better than the latency that every process running in your\nenvironment will experience because of the kernel or hypervisor implementation\nor setup.\n\nWe call this kind of latency **intrinsic latency**, and `redis-cli` starting\nfrom Redis version 2.8.7 is able to measure it. This is an example run\nunder Linux 3.11.0 running on an entry level server.\n\nNote: the argument `100` is the number of seconds the test will be executed.\nThe more time we run the test, the more likely we'll be able to spot\nlatency spikes. 100 seconds is usually appropriate, however you may want\nto perform a few runs at different times. Please note that the test is CPU\nintensive and will likely saturate a single core in your system.\n\n    $ ./redis-cli --intrinsic-latency 100\n    Max latency so far: 1 microseconds.\n    Max latency so far: 16 microseconds.\n    Max latency so far: 50 microseconds.\n    Max latency so far: 53 microseconds.\n    Max latency so far: 83 microseconds.\n    Max latency so far: 115 microseconds.\n\nNote: redis-cli in this special case needs to **run in the server** where you run or plan to run Redis, not in the client. In this special mode redis-cli does not connect to a Redis server at all: it will just try to measure the largest time the kernel does not provide CPU time to run to the redis-cli process itself.\n\nIn the above example, the intrinsic latency of the system is just 0.115\nmilliseconds (or 115 microseconds), which is a good news, however keep in mind\nthat the intrinsic latency may change over time depending on the load of the\nsystem.\n\nVirtualized environments will not show so good numbers, especially with high\nload or if there are noisy neighbors. The following is a run on a Linode 4096\ninstance running Redis and Apache:\n\n    $ ./redis-cli --intrinsic-latency 100\n    Max latency so far: 573 microseconds.\n    Max latency so far: 695 microseconds.\n    Max latency so far: 919 microseconds.\n    Max latency so far: 1606 microseconds.\n    Max latency so far: 3191 microseconds.\n    Max latency so far: 9243 microseconds.\n    Max latency so far: 9671 microseconds.\n\nHere we have an intrinsic latency of 9.7 milliseconds: this means that we can't ask better than that to Redis. However other runs at different times in different virtualization environments with higher load or with noisy neighbors can easily show even worse values. We were able to measure up to 40 milliseconds in\nsystems otherwise apparently running normally.\n\nLatency induced by network and communication\n--------------------------------------------\n\nClients connect to Redis using a TCP/IP connection or a Unix domain connection.\nThe typical latency of a 1 Gbit/s network is about 200 us, while the latency\nwith a Unix domain socket can be as low as 30 us. It actually depends on your\nnetwork and system hardware. On top of the communication itself, the system\nadds some more latency (due to thread scheduling, CPU caches, NUMA placement,\netc ...). System induced latencies are significantly higher on a virtualized\nenvironment than on a physical machine.\n\nThe consequence is even if Redis processes most commands in sub microsecond\nrange, a client performing many roundtrips to the server will have to pay\nfor these network and system related latencies.\n\nAn efficient client will therefore try to limit the number of roundtrips by\npipelining several commands together. This is fully supported by the servers\nand most clients. Aggregated commands like MSET/MGET can be also used for\nthat purpose. Starting with Redis 2.4, a number of commands also support\nvariadic parameters for all data types.\n\nHere are some guidelines:\n\n+ If you can afford it, prefer a physical machine over a VM to host the server.\n+ Do not systematically connect/disconnect to the server (especially true\n  for web based applications). Keep your connections as long lived as possible.\n+ If your client is on the same host than the server, use Unix domain sockets.\n+ Prefer to use aggregated commands (MSET/MGET), or commands with variadic\n  parameters (if possible) over pipelining.\n+ Prefer to use pipelining (if possible) over sequence of roundtrips.\n+ Redis supports Lua server-side scripting to cover cases that are not suitable\n  for raw pipelining (for instance when the result of a command is an input for\n  the following commands).\n\nOn Linux, some people can achieve better latencies by playing with process\nplacement (taskset), cgroups, real-time priorities (chrt), NUMA\nconfiguration (numactl), or by using a low-latency kernel. Please note\nvanilla Redis is not really suitable to be bound on a **single** CPU core.\nRedis can fork background tasks that can be extremely CPU consuming\nlike [`BGSAVE`](https://redis.io/docs/latest/commands/bgsave) or [`BGREWRITEAOF`](https://redis.io/docs/latest/commands/bgrewriteaof). These tasks must **never** run on the same core\nas the main event loop.\n\nIn most situations, these kind of system level optimizations are not needed.\nOnly do them if you require them, and if you are familiar with them.\n\nSingle threaded nature of Redis\n-------------------------------\n\nRedis uses a *mostly* single threaded design. This means that a single process\nserves all the client requests, using a technique called **multiplexing**.\nThis means that Redis can serve a single request in every given moment, so\nall the requests are served sequentially. This is very similar to how Node.js\nworks as well. However, both products are not often perceived as being slow.\nThis is caused in part by the small amount of time to complete a single request,\nbut primarily because these products are designed to not block on system calls,\nsuch as reading data from or writing data to a socket.\n\nI said that Redis is *mostly* single threaded since actually from Redis 2.4\nwe use threads in Redis in order to perform some slow I/O operations in the\nbackground, mainly related to disk I/O, but this does not change the fact\nthat Redis serves all the requests using a single thread.\n\nLatency generated by slow commands\n----------------------------------\n\nA consequence of being single thread is that when a request is slow to serve\nall the other clients will wait for this request to be served. When executing\nnormal commands, like [`GET`](https://redis.io/docs/latest/commands/get) or [`SET`](https://redis.io/docs/latest/commands/set) or [`LPUSH`](https://redis.io/docs/latest/commands/lpush) this is not a problem\nat all since these commands are executed in constant (and very small) time.\nHowever there are commands operating on many elements, like [`SORT`](https://redis.io/docs/latest/commands/sort), [`LREM`](https://redis.io/docs/latest/commands/lrem),\n[`SUNION`](https://redis.io/docs/latest/commands/sunion) and others. For instance taking the intersection of two big sets\ncan take a considerable amount of time.\n\nThe algorithmic complexity of all commands is documented. A good practice\nis to systematically check it when using commands you are not familiar with.\n\nIf you have latency concerns you should either not use slow commands against\nvalues composed of many elements, or you should run a replica using Redis\nreplication where you run all your slow queries.\n\nIt is possible to monitor slow commands using the Redis\n[Slow Log feature](https://redis.io/docs/latest/commands/slowlog).\n\nAdditionally, you can use your favorite per-process monitoring program\n(top, htop, prstat, etc ...) to quickly check the CPU consumption of the\nmain Redis process. If it is high while the traffic is not, it is usually\na sign that slow commands are used.\n\n**IMPORTANT NOTE**: a VERY common source of latency generated by the execution\nof slow commands is the use of the [`KEYS`](https://redis.io/docs/latest/commands/keys) command in production environments.\n[`KEYS`](https://redis.io/docs/latest/commands/keys), as documented in the Redis documentation, should only be used for\ndebugging purposes. Since Redis 2.8 a new commands were introduced in order to\niterate the key space and other large collections incrementally, please check\nthe [`SCAN`](https://redis.io/docs/latest/commands/scan), [`SSCAN`](https://redis.io/docs/latest/commands/sscan), [`HSCAN`](https://redis.io/docs/latest/commands/hscan) and [`ZSCAN`](https://redis.io/docs/latest/commands/zscan) commands for more information.\n\nLatency generated by fork\n-------------------------\n\nIn order to generate the RDB file in background, or to rewrite the Append Only File if AOF persistence is enabled, Redis has to fork background processes.\nThe fork operation (running in the main thread) can induce latency by itself.\n\nForking is an expensive operation on most Unix-like systems, since it involves\ncopying a good number of objects linked to the process. This is especially\ntrue for the page table associated to the virtual memory mechanism.\n\nFor instance on a Linux/AMD64 system, the memory is divided in 4 kB pages.\nTo convert virtual addresses to physical addresses, each process stores\na page table (actually represented as a tree) containing at least a pointer\nper page of the address space of the process. So a large 24 GB Redis instance\nrequires a page table of 24 GB / 4 kB * 8 = 48 MB.\n\nWhen a background save is performed, this instance will have to be forked,\nwhich will involve allocating and copying 48 MB of memory. It takes time\nand CPU, especially on virtual machines where allocation and initialization\nof a large memory chunk can be expensive.\n\nFork time in different systems\n------------------------------\n\nModern hardware is pretty fast at copying the page table, but Xen is not.\nThe problem with Xen is not virtualization-specific, but Xen-specific. For instance using VMware or Virtual Box does not result into slow fork time.\nThe following is a table that compares fork time for different Redis instance\nsize. Data is obtained performing a BGSAVE and looking at the `latest_fork_usec` filed in the [`INFO`](https://redis.io/docs/latest/commands/info) command output.\n\nHowever the good news is that **new types of EC2 HVM based instances are much\nbetter with fork times**, almost on par with physical servers, so for example\nusing m3.medium (or better) instances will provide good results.\n\n* **Linux beefy VM on VMware** 6.0GB RSS forked in 77 milliseconds (12.8 milliseconds per GB).\n* **Linux running on physical machine (Unknown HW)** 6.1GB RSS forked in 80 milliseconds (13.1 milliseconds per GB)\n* **Linux running on physical machine (Xeon @ 2.27Ghz)** 6.9GB RSS forked into 62 milliseconds (9 milliseconds per GB).\n* **Linux VM on 6sync (KVM)** 360 MB RSS forked in 8.2 milliseconds (23.3 milliseconds per GB).\n* **Linux VM on EC2, old instance types (Xen)** 6.1GB RSS forked in 1460 milliseconds (239.3 milliseconds per GB).\n* **Linux VM on EC2, new instance types (Xen)** 1GB RSS forked in 10 milliseconds (10 milliseconds per GB).\n* **Linux VM on Linode (Xen)** 0.9GBRSS forked into 382 milliseconds (424 milliseconds per GB).\n\nAs you can see certain VMs running on Xen have a performance hit that is between one order to two orders of magnitude. For EC2 users the suggestion is simple: use modern HVM based instances.\n\nLatency induced by transparent huge pages\n-----------------------------------------\n\nUnfortunately when a Linux kernel has transparent huge pages enabled, Redis\nincurs to a big latency penalty after the `fork` call is used in order to\npersist on disk. Huge pages are the cause of the following issue:\n\n1. Fork is called, two processes with shared huge pages are created.\n2. In a busy instance, a few event loops runs will cause commands to target a few thousand of pages, causing the copy on write of almost the whole process memory.\n3. This will result in big latency and big memory usage.\n\nMake sure to **disable transparent huge pages** using the following command:\n\n    echo never > /sys/kernel/mm/transparent_hugepage/enabled\n\nLatency induced by swapping (operating system paging)\n-----------------------------------------------------\n\nLinux (and many other modern operating systems) is able to relocate memory\npages from the memory to the disk, and vice versa, in order to use the\nsystem memory efficiently.\n\nIf a Redis page is moved by the kernel from the memory to the swap file, when\nthe data stored in this memory page is used by Redis (for example accessing\na key stored into this memory page) the kernel will stop the Redis process\nin order to move the page back into the main memory. This is a slow operation\ninvolving random I/Os (compared to accessing a page that is already in memory)\nand will result into anomalous latency experienced by Redis clients.\n\nThe kernel relocates Redis memory pages on disk mainly because of three reasons:\n\n* The system is under memory pressure since the running processes are demanding\nmore physical memory than the amount that is available. The simplest instance of\nthis problem is simply Redis using more memory than is available.\n* The Redis instance data set, or part of the data set, is mostly completely idle\n(never accessed by clients), so the kernel could swap idle memory pages on disk.\nThis problem is very rare since even a moderately slow instance will touch all\nthe memory pages often, forcing the kernel to retain all the pages in memory.\n* Some processes are generating massive read or write I/Os on the system. Because\nfiles are generally cached, it tends to put pressure on the kernel to increase\nthe filesystem cache, and therefore generate swapping activity. Please note it\nincludes Redis RDB and/or AOF background threads which can produce large files.\n\nFortunately Linux offers good tools to investigate the problem, so the simplest\nthing to do is when latency due to swapping is suspected is just to check if\nthis is the case.\n\nThe first thing to do is to checking the amount of Redis memory that is swapped\non disk. In order to do so you need to obtain the Redis instance pid:\n\n    $ redis-cli info | grep process_id\n    process_id:5454\n\nNow enter the /proc file system directory for this process:\n\n    $ cd /proc/5454\n\nHere you'll find a file called **smaps** that describes the memory layout of\nthe Redis process (assuming you are using Linux 2.6.16 or newer).\nThis file contains very detailed information about our process memory maps,\nand one field called **Swap** is exactly what we are looking for. However\nthere is not just a single swap field since the smaps file contains the\ndifferent memory maps of our Redis process (The memory layout of a process\nis more complex than a simple linear array of pages).\n\nSince we are interested in all the memory swapped by our process the first thing\nto do is to grep for the Swap field across all the file:\n\n    $ cat smaps | grep 'Swap:'\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                 12 kB\n    Swap:                156 kB\n    Swap:                  8 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  4 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  4 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  4 kB\n    Swap:                  4 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n    Swap:                  0 kB\n\nIf everything is 0 kB, or if there are sporadic 4k entries, everything is\nperfectly normal. Actually in our example instance (the one of a real web\nsite running Redis and serving hundreds of users every second) there are a\nfew entries that show more swapped pages. To investigate if this is a serious\nproblem or not we change our command in order to also print the size of the\nmemory map:\n\n    $ cat smaps | egrep '^(Swap|Size)'\n    Size:                316 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  8 kB\n    Swap:                  0 kB\n    Size:                 40 kB\n    Swap:                  0 kB\n    Size:                132 kB\n    Swap:                  0 kB\n    Size:             720896 kB\n    Swap:                 12 kB\n    Size:               4096 kB\n    Swap:                156 kB\n    Size:               4096 kB\n    Swap:                  8 kB\n    Size:               4096 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:               1272 kB\n    Swap:                  0 kB\n    Size:                  8 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                 16 kB\n    Swap:                  0 kB\n    Size:                 84 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  8 kB\n    Swap:                  4 kB\n    Size:                  8 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  4 kB\n    Size:                144 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  4 kB\n    Size:                 12 kB\n    Swap:                  4 kB\n    Size:                108 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n    Size:                272 kB\n    Swap:                  0 kB\n    Size:                  4 kB\n    Swap:                  0 kB\n\nAs you can see from the output, there is a map of 720896 kB\n(with just 12 kB swapped) and 156 kB more swapped in another map:\nbasically a very small amount of our memory is swapped so this is not\ngoing to create any problem at all.\n\nIf instead a non trivial amount of the process memory is swapped on disk your\nlatency problems are likely related to swapping. If this is the case with your\nRedis instance you can further verify it using the **vmstat** command:\n\n    $ vmstat 1\n    procs -----------memory---------- ---swap-- -----io---- -system-- ----cpu----\n     r  b   swpd   free   buff  cache   si   so    bi    bo   in   cs us sy id wa\n     0  0   3980 697932 147180 1406456    0    0     2     2    2    0  4  4 91  0\n     0  0   3980 697428 147180 1406580    0    0     0     0 19088 16104  9  6 84  0\n     0  0   3980 697296 147180 1406616    0    0     0    28 18936 16193  7  6 87  0\n     0  0   3980 697048 147180 1406640    0    0     0     0 18613 15987  6  6 88  0\n     2  0   3980 696924 147180 1406656    0    0     0     0 18744 16299  6  5 88  0\n     0  0   3980 697048 147180 1406688    0    0     0     4 18520 15974  6  6 88  0\n    ^C\n\nThe interesting part of the output for our needs are the two columns **si**\nand **so**, that counts the amount of memory swapped from/to the swap file. If\nyou see non zero counts in those two columns then there is swapping activity\nin your system.\n\nFinally, the **iostat** command can be used to check the global I/O activity of\nthe system.\n\n    $ iostat -xk 1\n    avg-cpu:  %user   %nice %system %iowait  %steal   %idle\n              13.55    0.04    2.92    0.53    0.00   82.95\n\n    Device:         rrqm/s   wrqm/s     r/s     w/s    rkB/s    wkB/s avgrq-sz avgqu-sz   await  svctm  %util\n    sda               0.77     0.00    0.01    0.00     0.40     0.00    73.65     0.00    3.62   2.58   0.00\n    sdb               1.27     4.75    0.82    3.54    38.00    32.32    32.19     0.11   24.80   4.24   1.85\n\nIf your latency problem is due to Redis memory being swapped on disk you need\nto lower the memory pressure in your system, either adding more RAM if Redis\nis using more memory than the available, or avoiding running other memory\nhungry processes in the same system.\n\nLatency due to AOF and disk I/O\n-------------------------------\n\nAnother source of latency is due to the Append Only File support on Redis.\nThe AOF basically uses two system calls to accomplish its work. One is\nwrite(2) that is used in order to write data to the append only file, and\nthe other one is fdatasync(2) that is used in order to flush the kernel\nfile buffer on disk in order to ensure the durability level specified by\nthe user.\n\nBoth the write(2) and fdatasync(2) calls can be source of latency.\nFor instance write(2) can block both when there is a system wide sync\nin progress, or when the output buffers are full and the kernel requires\nto flush on disk in order to accept new writes.\n\nThe fdatasync(2) call is a worse source of latency as with many combinations\nof kernels and file systems used it can take from a few milliseconds to\na few seconds to complete, especially in the case of some other process\ndoing I/O. For this reason when possible Redis does the fdatasync(2) call\nin a different thread since Redis 2.4.\n\nWe'll see how configuration can affect the amount and source of latency\nwhen using the AOF file.\n\nThe AOF can be configured to perform a fsync on disk in three different\nways using the **appendfsync** configuration option (this setting can be\nmodified at runtime using the **CONFIG SET** command).\n\n* When appendfsync is set to the value of **no** Redis performs no fsync.\nIn this configuration the only source of latency can be write(2).\nWhen this happens usually there is no solution since simply the disk can't\ncope with the speed at which Redis is receiving data, however this is\nuncommon if the disk is not seriously slowed down by other processes doing\nI/O.\n\n* When appendfsync is set to the value of **everysec** Redis performs a\nfsync every second. It uses a different thread, and if the fsync is still\nin progress Redis uses a buffer to delay the write(2) call up to two seconds\n(since write would block on Linux if a fsync is in progress against the\nsame file). However if the fsync is taking too long Redis will eventually\nperform the write(2) call even if the fsync is still in progress, and this\ncan be a source of latency.\n\n* When appendfsync is set to the value of **always** a fsync is performed\nat every write operation before replying back to the client with an OK code\n(actually Redis will try to cluster many commands executed at the same time\ninto a single fsync). In this mode performances are very low in general and\nit is strongly recommended to use a fast disk and a file system implementation\nthat can perform the fsync in short time.\n\nMost Redis users will use either the **no** or **everysec** setting for the\nappendfsync configuration directive. The suggestion for minimum latency is\nto avoid other processes doing I/O in the same system.\nUsing an SSD disk can help as well, but usually even non SSD disks perform\nwell with the append only file if the disk is spare as Redis writes\nto the append only file without performing any seek.\n\nIf you want to investigate your latency issues related to the append only\nfile you can use the strace command under Linux:\n\n    sudo strace -p $(pidof redis-server) -T -e trace=fdatasync\n\nThe above command will show all the fdatasync(2) system calls performed by\nRedis in the main thread. With the above command you'll not see the\nfdatasync system calls performed by the background thread when the\nappendfsync config option is set to **everysec**. In order to do so\njust add the -f switch to strace.\n\nIf you wish you can also see both fdatasync and write system calls with the\nfollowing command:\n\n    sudo strace -p $(pidof redis-server) -T -e trace=fdatasync,write\n\nHowever since write(2) is also used in order to write data to the client\nsockets this will likely show too many things unrelated to disk I/O.\nApparently there is no way to tell strace to just show slow system calls so\nI use the following command:\n\n    sudo strace -f -p $(pidof redis-server) -T -e trace=fdatasync,write 2>&1 | grep -v '0.0' | grep -v unfinished\n\nLatency generated by expires\n----------------------------\n\nRedis evict expired keys in two ways:\n\n+ One *lazy* way expires a key when it is requested by a command, but it is found to be already expired.\n+ One *active* way expires a few keys every 100 milliseconds.\n\nThe active expiring is designed to be adaptive. An expire cycle is started every 100 milliseconds (10 times per second), and will do the following:\n\n+ Sample `ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP` keys, evicting all the keys already expired.\n+ If the more than 25% of the keys were found expired, repeat.\n\nGiven that `ACTIVE_EXPIRE_CYCLE_LOOKUPS_PER_LOOP` is set to 20 by default, and the process is performed ten times per second, usually just 200 keys per second are actively expired. This is enough to clean the DB fast enough even when already expired keys are not accessed for a long time, so that the *lazy* algorithm does not help. At the same time expiring just 200 keys per second has no effects in the latency a Redis instance.\n\nHowever the algorithm is adaptive and will loop if it finds more than 25% of keys already expired in the set of sampled keys. But given that we run the algorithm ten times per second, this means that the unlucky event of more than 25% of the keys in our random sample are expiring at least *in the same second*.\n\nBasically this means that **if the database has many, many keys expiring in the same second, and these make up at least 25% of the current population of keys with an expire set**, Redis can block in order to get the percentage of keys already expired below 25%.\n\nThis approach is needed in order to avoid using too much memory for keys that are already expired, and usually is absolutely harmless since it's strange that a big number of keys are going to expire in the same exact second, but it is not impossible that the user used [`EXPIREAT`](https://redis.io/docs/latest/commands/expireat) extensively with the same Unix time.\n\nIn short: be aware that many keys expiring at the same moment can be a source of latency.\n\nRedis software watchdog\n---\n\nRedis 2.6 introduces the *Redis Software Watchdog* that is a debugging tool\ndesigned to track those latency problems that for one reason or the other\nescaped an analysis using normal tools.\n\nThe software watchdog is an experimental feature. While it is designed to\nbe used in production environments care should be taken to backup the database\nbefore proceeding as it could possibly have unexpected interactions with the\nnormal execution of the Redis server.\n\nIt is important to use it only as *last resort* when there is no way to track the issue by other means.\n\nThis is how this feature works:\n\n* The user enables the software watchdog using the [`CONFIG SET`](https://redis.io/docs/latest/commands/config-set) command.\n* Redis starts monitoring itself constantly.\n* If Redis detects that the server is blocked into some operation that is not returning fast enough, and that may be the source of the latency issue, a low level report about where the server is blocked is dumped on the log file.\n* The user contacts the developers writing a message in the Redis Google Group, including the watchdog report in the message.\n\nNote that this feature cannot be enabled using the redis.conf file, because it is designed to be enabled only in already running instances and only for debugging purposes.\n\nTo enable the feature just use the following:\n\n    CONFIG SET watchdog-period 500\n\nThe period is specified in milliseconds. In the above example I specified to log latency issues only if the server detects a delay of 500 milliseconds or greater. The minimum configurable period is 200 milliseconds.\n\nWhen you are done with the software watchdog you can turn it off setting the `watchdog-period` parameter to 0. **Important:** remember to do this because keeping the instance with the watchdog turned on for a longer time than needed is generally not a good idea.\n\nThe following is an example of what you'll see printed in the log file once the software watchdog detects a delay longer than the configured one:\n\n    [8547 | signal handler] (1333114359)\n    --- WATCHDOG TIMER EXPIRED ---\n    /lib/libc.so.6(nanosleep+0x2d) [0x7f16b5c2d39d]\n    /lib/libpthread.so.0(+0xf8f0) [0x7f16b5f158f0]\n    /lib/libc.so.6(nanosleep+0x2d) [0x7f16b5c2d39d]\n    /lib/libc.so.6(usleep+0x34) [0x7f16b5c62844]\n    ./redis-server(debugCommand+0x3e1) [0x43ab41]\n    ./redis-server(call+0x5d) [0x415a9d]\n    ./redis-server(processCommand+0x375) [0x415fc5]\n    ./redis-server(processInputBuffer+0x4f) [0x4203cf]\n    ./redis-server(readQueryFromClient+0xa0) [0x4204e0]\n    ./redis-server(aeProcessEvents+0x128) [0x411b48]\n    ./redis-server(aeMain+0x2b) [0x411dbb]\n    ./redis-server(main+0x2b6) [0x418556]\n    /lib/libc.so.6(__libc_start_main+0xfd) [0x7f16b5ba1c4d]\n    ./redis-server() [0x411099]\n    ------\n\nNote: in the example the **DEBUG SLEEP** command was used in order to block the server. The stack trace is different if the server blocks in a different context.\n\nIf you happen to collect multiple watchdog stack traces you are encouraged to send everything to the Redis Google Group: the more traces we obtain, the simpler it will be to understand what the problem with your instance is."
    }
  ],
  "examples": []
}