{
  "schema_version": 2,
  "id": "operate/oss_and_stack/management/sentinel",
  "title": "High availability with Redis Sentinel",
  "url": "https://redis.io/docs/latest/operate/oss_and_stack/management/sentinel/",
  "summary": "High availability for non-clustered Redis",
  "aliases": [
    "/management/sentinel/",
    "/manual/sentinel/"
  ],
  "tags": [
    "docs",
    "operate",
    "stack",
    "oss"
  ],
  "last_updated": "2026-10-13T16:38:58+03:00",
  "page_type": "content",
  "content_hash": "4648fec915414c1774ca81f807b9db6559968a71167e3c9f2a3f5c332c0d0326",
  "sections": [
    {
      "id": "overview",
      "title": "Overview",
      "role": "overview",
      "text": "Redis Sentinel provides high availability for Redis when not using [Redis Cluster](https://redis.io/docs/latest/operate/oss_and_stack/management/scaling). \n\nRedis Sentinel also provides other collateral tasks such as monitoring,\nnotifications and acts as a configuration provider for clients.\n\nThis is the full list of Sentinel capabilities at a macroscopic level (i.e. the *big picture*):\n\n* **Monitoring**. Sentinel constantly checks if your master and replica instances are working as expected.\n* **Notification**. Sentinel can notify the system administrator, or other computer programs, via an API, that something is wrong with one of the monitored Redis instances.\n* **Automatic failover**. If a master is not working as expected, Sentinel can start a failover process where a replica is promoted to master, the other additional replicas are reconfigured to use the new master, and the applications using the Redis server are informed about the new address to use when connecting.\n* **Configuration provider**. Sentinel acts as a source of authority for clients service discovery: clients connect to Sentinels in order to ask for the address of the current Redis master responsible for a given service. If a failover occurs, Sentinels will report the new address."
    },
    {
      "id": "sentinel-as-a-distributed-system",
      "title": "Sentinel as a distributed system",
      "role": "content",
      "text": "Redis Sentinel is a distributed system:\n\nSentinel itself is designed to run in a configuration where there are multiple Sentinel processes cooperating together. The advantage of having multiple Sentinel processes cooperating are the following:\n\n1. Failure detection is performed when multiple Sentinels agree about the fact a given master is no longer available. This lowers the probability of false positives.\n2. Sentinel works even if not all the Sentinel processes are working, making the system robust against failures. There is no fun in having a failover system which is itself a single point of failure, after all.\n\nThe sum of Sentinels, Redis instances (masters and replicas) and clients\nconnecting to Sentinel and Redis, are also a larger distributed system with\nspecific properties. In this document concepts will be introduced gradually\nstarting from basic information needed in order to understand the basic\nproperties of Sentinel, to more complex information (that are optional) in\norder to understand how exactly Sentinel works."
    },
    {
      "id": "sentinel-quick-start",
      "title": "Sentinel quick start",
      "role": "content",
      "text": ""
    },
    {
      "id": "obtaining-sentinel",
      "title": "Obtaining Sentinel",
      "role": "content",
      "text": "The current version of Sentinel is called **Sentinel 2**. It is a rewrite of\nthe initial Sentinel implementation using stronger and simpler-to-predict\nalgorithms (that are explained in this documentation).\n\nA stable release of Redis Sentinel is shipped since Redis 2.8.\n\nNew developments are performed in the *unstable* branch, and new features\nsometimes are back ported into the latest stable branch as soon as they are\nconsidered to be stable.\n\nRedis Sentinel version 1, shipped with Redis 2.6, is deprecated and should not be used."
    },
    {
      "id": "running-sentinel",
      "title": "Running Sentinel",
      "role": "content",
      "text": "If you are using the `redis-sentinel` executable (or if you have a symbolic\nlink with that name to the `redis-server` executable) you can run Sentinel\nwith the following command line:\n\n    redis-sentinel /path/to/sentinel.conf\n\nOtherwise you can use directly the `redis-server` executable starting it in\nSentinel mode:\n\n    redis-server /path/to/sentinel.conf --sentinel\n\nBoth ways work the same.\n\nHowever **it is mandatory** to use a configuration file when running Sentinel, as this file will be used by the system in order to save the current state that will be reloaded in case of restarts. Sentinel will simply refuse to start if no configuration file is given or if the configuration file path is not writable.\n\nSentinels by default run **listening for connections to TCP port 26379**, so\nfor Sentinels to work, port 26379 of your servers **must be open** to receive\nconnections from the IP addresses of the other Sentinel instances.\nOtherwise Sentinels can't talk and can't agree about what to do, so failover\nwill never be performed."
    },
    {
      "id": "fundamental-things-to-know-about-sentinel-before-deploying",
      "title": "Fundamental things to know about Sentinel before deploying",
      "role": "content",
      "text": "1. You need at least three Sentinel instances for a robust deployment.\n2. The three Sentinel instances should be placed into computers or virtual machines that are believed to fail in an independent way. So for example different physical servers or Virtual Machines executed on different availability zones.\n3. Sentinel + Redis distributed system does not guarantee that acknowledged writes are retained during failures, since Redis uses asynchronous replication. However there are ways to deploy Sentinel that make the window to lose writes limited to certain moments, while there are other less secure ways to deploy it.\n4. You need Sentinel support in your clients. Popular client libraries have Sentinel support, but not all.\n5. There is no HA setup which is safe if you don't test from time to time in development environments, or even better if you can, in production environments, if they work. You may have a misconfiguration that will become apparent only when it's too late (at 3am when your master stops working).\n6. **Sentinel, Docker, or other forms of Network Address Translation or Port Mapping should be mixed with care**: Docker performs port remapping, breaking Sentinel auto discovery of other Sentinel processes and the list of replicas for a master. Check the [section about _Sentinel and Docker_](#sentinel-docker-nat-and-possible-issues) later in this document for more information."
    },
    {
      "id": "configuring-sentinel",
      "title": "Configuring Sentinel",
      "role": "configuration",
      "text": "The Redis source distribution contains a file called `sentinel.conf`\nthat is a self-documented example configuration file you can use to\nconfigure Sentinel, however a typical minimal configuration file looks like the\nfollowing:\n\n    sentinel monitor mymaster 127.0.0.1 6379 2\n    sentinel down-after-milliseconds mymaster 60000\n    sentinel failover-timeout mymaster 180000\n    sentinel parallel-syncs mymaster 1\n\n    sentinel monitor resque 192.168.1.3 6380 4\n    sentinel down-after-milliseconds resque 10000\n    sentinel failover-timeout resque 180000\n    sentinel parallel-syncs resque 5\n\nYou only need to specify the masters to monitor, giving to each separated\nmaster (that may have any number of replicas) a different name. There is no\nneed to specify replicas, which are auto-discovered. Sentinel will update the\nconfiguration automatically with additional information about replicas (in\norder to retain the information in case of restart). The configuration is\nalso rewritten every time a replica is promoted to master during a failover\nand every time a new Sentinel is discovered.\n\nThe example configuration above basically monitors two sets of Redis\ninstances, each composed of a master and an undefined number of replicas.\nOne set of instances is called `mymaster`, and the other `resque`.\n\nThe meaning of the arguments of `sentinel monitor` statements is the following:\n\n    sentinel monitor <master-name> <ip> <port> <quorum>\n\nFor the sake of clarity, let's check line by line what the configuration\noptions mean:\n\nThe first line is used to tell Redis to monitor a master called *mymaster*,\nthat is at address 127.0.0.1 and port 6379, with a quorum of 2. Everything\nis pretty obvious but the **quorum** argument:\n\n* The **quorum** is the number of Sentinels that need to agree about the fact the master is not reachable, in order to really mark the master as failing, and eventually start a failover procedure if possible.\n* However **the quorum is only used to detect the failure**. In order to actually perform a failover, one of the Sentinels need to be elected leader for the failover and be authorized to proceed. This only happens with the vote of the **majority of the Sentinel processes**.\n\nSo for example if you have 5 Sentinel processes, and the quorum for a given\nmaster set to the value of 2, this is what happens:\n\n* If two Sentinels agree at the same time about the master being unreachable, one of the two will try to start a failover.\n* If there are at least a total of three Sentinels reachable, the failover will be authorized and will actually start.\n\nIn practical terms this means during failures **Sentinel never starts a failover if the majority of Sentinel processes are unable to talk** (aka no failover in the minority partition)."
    },
    {
      "id": "other-sentinel-options",
      "title": "Other Sentinel options",
      "role": "content",
      "text": "The other options are almost always in the form:\n\n    sentinel <option_name> <master_name> <option_value>\n\nAnd are used for the following purposes:\n\n* `down-after-milliseconds` is the time in milliseconds an instance should not\nbe reachable (either does not reply to our PINGs or it is replying with an\nerror) for a Sentinel starting to think it is down.\n* `parallel-syncs` sets the number of replicas that can be reconfigured to use\nthe new master after a failover at the same time. The lower the number, the\nmore time it will take for the failover process to complete, however if the\nreplicas are configured to serve old data, you may not want all the replicas to\nre-synchronize with the master at the same time. While the replication\nprocess is mostly non blocking for a replica, there is a moment when it stops to\nload the bulk data from the master. You may want to make sure only one replica\nat a time is not reachable by setting this option to the value of 1.\n\nAdditional options are described in the rest of this document and\ndocumented in the example `sentinel.conf` file shipped with the Redis\ndistribution.\n\nConfiguration parameters can be modified at runtime:\n\n* Master-specific configuration parameters are modified using `SENTINEL SET`.\n* Global configuration parameters are modified using `SENTINEL CONFIG SET`.\n\nSee the [_Reconfiguring Sentinel at runtime_ section](#reconfiguring-sentinel-at-runtime) for more information."
    },
    {
      "id": "example-sentinel-deployments",
      "title": "Example Sentinel deployments",
      "role": "example",
      "text": "Now that you know the basic information about Sentinel, you may wonder where\nyou should place your Sentinel processes, how many Sentinel processes you need\nand so forth. This section shows a few example deployments.\n\nWe use ASCII art in order to show you configuration examples in a *graphical*\nformat, this is what the different symbols means:\n\n    +--------------------+\n    | This is a computer |\n    | or VM that fails   |\n    | independently. We  |\n    | call it a \"box\"    |\n    +--------------------+\n\nWe write inside the boxes what they are running:\n\n    +-------------------+\n    | Redis master M1   |\n    | Redis Sentinel S1 |\n    +-------------------+\n\nDifferent boxes are connected by lines, to show that they are able to talk:\n\n    +-------------+               +-------------+\n    | Sentinel S1 |---------------| Sentinel S2 |\n    +-------------+               +-------------+\n\nNetwork partitions are shown as interrupted lines using slashes:\n\n    +-------------+                +-------------+\n    | Sentinel S1 |------ // ------| Sentinel S2 |\n    +-------------+                +-------------+\n\nAlso note that:\n\n* Masters are called M1, M2, M3, ..., Mn.\n* Replicas are called R1, R2, R3, ..., Rn (R stands for *replica*).\n* Sentinels are called S1, S2, S3, ..., Sn.\n* Clients are called C1, C2, C3, ..., Cn.\n* When an instance changes role because of Sentinel actions, we put it inside square brackets, so [M1] means an instance that is now a master because of Sentinel intervention.\n\nNote that we will never show **setups where just two Sentinels are used**, since\nSentinels always need **to talk with the majority** in order to start a\nfailover.\n\n#### Example 1: just two Sentinels, DON'T DO THIS\n\n    +----+         +----+\n    | M1 |---------| R1 |\n    | S1 |         | S2 |\n    +----+         +----+\n\n    Configuration: quorum = 1\n\n* In this setup, if the master M1 fails, R1 will be promoted since the two Sentinels can reach agreement about the failure (obviously with quorum set to 1) and can also authorize a failover because the majority is two. So apparently it could superficially work, however check the next points to see why this setup is broken.\n* If the box where M1 is running stops working, also S1 stops working. The Sentinel running in the other box S2 will not be able to authorize a failover, so the system will become not available.\n\nNote that a majority is needed in order to order different failovers, and later propagate the latest configuration to all the Sentinels. Also note that the ability to failover in a single side of the above setup, without any agreement, would be very dangerous:\n\n    +----+           +------+\n    | M1 |----//-----| [M1] |\n    | S1 |           | S2   |\n    +----+           +------+\n\nIn the above configuration we created two masters (assuming S2 could failover\nwithout authorization) in a perfectly symmetrical way. Clients may write\nindefinitely to both sides, and there is no way to understand when the\npartition heals what configuration is the right one, in order to prevent\na *permanent split brain condition*.\n\nSo please **deploy at least three Sentinels in three different boxes** always.\n\n#### Example 2: basic setup with three boxes\n\nThis is a very simple setup, that has the advantage to be simple to tune\nfor additional safety. It is based on three boxes, each box running both\na Redis process and a Sentinel process.\n\n\n           +----+\n           | M1 |\n           | S1 |\n           +----+\n              |\n    +----+    |    +----+\n    | R2 |----+----| R3 |\n    | S2 |         | S3 |\n    +----+         +----+\n\n    Configuration: quorum = 2\n\nIf the master M1 fails, S2 and S3 will agree about the failure and will\nbe able to authorize a failover, making clients able to continue.\n\nIn every Sentinel setup, as Redis uses asynchronous replication, there is\nalways the risk of losing some writes because a given acknowledged write\nmay not be able to reach the replica which is promoted to master. However in\nthe above setup there is a higher risk due to clients being partitioned away\nwith an old master, like in the following picture:\n\n             +----+\n             | M1 |\n             | S1 | <- C1 (writes will be lost)\n             +----+\n                |\n                /\n                /\n    +------+    |    +----+\n    | [M2] |----+----| R3 |\n    | S2   |         | S3 |\n    +------+         +----+\n\nIn this case a network partition isolated the old master M1, so the\nreplica R2 is promoted to master. However clients, like C1, that are\nin the same partition as the old master, may continue to write data\nto the old master. This data will be lost forever since when the partition\nwill heal, the master will be reconfigured as a replica of the new master,\ndiscarding its data set.\n\nThis problem can be mitigated using the following Redis replication\nfeature, that allows to stop accepting writes if a master detects that\nit is no longer able to transfer its writes to the specified number of replicas.\n\n    min-replicas-to-write 1\n    min-replicas-max-lag 10\n\nWith the above configuration (please see the self-commented `redis.conf` example in the Redis distribution for more information) a Redis instance, when acting as a master, will stop accepting writes if it can't write to at least 1 replica. Since replication is asynchronous *not being able to write* actually means that the replica is either disconnected, or is not sending us asynchronous acknowledges for more than the specified `max-lag` number of seconds.\n\nUsing this configuration, the old Redis master M1 in the above example, will become unavailable after 10 seconds. When the partition heals, the Sentinel configuration will converge to the new one, the client C1 will be able to fetch a valid configuration and will continue with the new master.\n\nHowever there is no free lunch. With this refinement, if the two replicas are\ndown, the master will stop accepting writes. It's a trade off.\n\n#### Example 3: Sentinel in the client boxes\n\nSometimes we have only two Redis boxes available, one for the master and\none for the replica. The configuration in the example 2 is not viable in\nthat case, so we can resort to the following, where Sentinels are placed\nwhere clients are:\n\n                +----+         +----+\n                | M1 |----+----| R1 |\n                |    |    |    |    |\n                +----+    |    +----+\n                          |\n             +------------+------------+\n             |            |            |\n             |            |            |\n          +----+        +----+      +----+\n          | C1 |        | C2 |      | C3 |\n          | S1 |        | S2 |      | S3 |\n          +----+        +----+      +----+\n\n          Configuration: quorum = 2\n\nIn this setup, the point of view Sentinels is the same as the clients: if\na master is reachable by the majority of the clients, it is fine.\nC1, C2, C3 here are generic clients, it does not mean that C1 identifies\na single client connected to Redis. It is more likely something like\nan application server, a Rails app, or something like that.\n\nIf the box where M1 and S1 are running fails, the failover will happen\nwithout issues, however it is easy to see that different network partitions\nwill result in different behaviors. For example Sentinel will not be able\nto setup if the network between the clients and the Redis servers is\ndisconnected, since the Redis master and replica will both be unavailable.\n\nNote that if C3 gets partitioned with M1 (hardly possible with\nthe network described above, but more likely possible with different\nlayouts, or because of failures at the software layer), we have a similar\nissue as described in Example 2, with the difference that here we have\nno way to break the symmetry, since there is just a replica and master, so\nthe master can't stop accepting queries when it is disconnected from its replica,\notherwise the master would never be available during replica failures.\n\nSo this is a valid setup but the setup in the Example 2 has advantages\nsuch as the HA system of Redis running in the same boxes as Redis itself\nwhich may be simpler to manage, and the ability to put a bound on the amount\nof time a master in the minority partition can receive writes.\n\n#### Example 4: Sentinel client side with less than three clients\n\nThe setup described in the Example 3 cannot be used if there are less than\nthree boxes in the client side (for example three web servers). In this\ncase we need to resort to a mixed setup like the following:\n\n                +----+         +----+\n                | M1 |----+----| R1 |\n                | S1 |    |    | S2 |\n                +----+    |    +----+\n                          |\n                   +------+-----+\n                   |            |\n                   |            |\n                +----+        +----+\n                | C1 |        | C2 |\n                | S3 |        | S4 |\n                +----+        +----+\n\n          Configuration: quorum = 3\n\nThis is similar to the setup in Example 3, but here we run four Sentinels\nin the four boxes we have available. If the master M1 becomes unavailable\nthe other three Sentinels will perform the failover.\n\nIn theory this setup works removing the box where C2 and S4 are running, and\nsetting the quorum to 2. However it is unlikely that we want HA in the\nRedis side without having high availability in our application layer."
    },
    {
      "id": "sentinel-docker-nat-and-possible-issues",
      "title": "Sentinel, Docker, NAT, and possible issues",
      "role": "content",
      "text": "Docker uses a technique called port mapping: programs running inside Docker\ncontainers may be exposed with a different port compared to the one the\nprogram believes to be using. This is useful in order to run multiple\ncontainers using the same ports, at the same time, in the same server.\n\nDocker is not the only software system where this happens, there are other\nNetwork Address Translation setups where ports may be remapped, and sometimes\nnot ports but also IP addresses.\n\nRemapping ports and addresses creates issues with Sentinel in two ways:\n\n1. Sentinel auto-discovery of other Sentinels no longer works, since it is based on *hello* messages where each Sentinel announce at which port and IP address they are listening for connection. However Sentinels have no way to understand that an address or port is remapped, so it is announcing an information that is not correct for other Sentinels to connect.\n2. Replicas are listed in the [`INFO`](https://redis.io/docs/latest/commands/info) output of a Redis master in a similar way: the address is detected by the master checking the remote peer of the TCP connection, while the port is advertised by the replica itself during the handshake, however the port may be wrong for the same reason as exposed in point 1.\n\nSince Sentinels auto detect replicas using masters [`INFO`](https://redis.io/docs/latest/commands/info) output information,\nthe detected replicas will not be reachable, and Sentinel will never be able to\nfailover the master, since there are no good replicas from the point of view of\nthe system, so there is currently no way to monitor with Sentinel a set of\nmaster and replica instances deployed with Docker, **unless you instruct Docker\nto map the port 1:1**.\n\nFor the first problem, in case you want to run a set of Sentinel\ninstances using Docker with forwarded ports (or any other NAT setup where ports\nare remapped), you can use the following two Sentinel configuration directives\nin order to force Sentinel to announce a specific set of IP and port:\n\n    sentinel announce-ip <ip>\n    sentinel announce-port <port>\n\nNote that Docker has the ability to run in *host networking mode* (check the `--net=host` option for more information). This should create no issues since ports are not remapped in this setup."
    },
    {
      "id": "ip-addresses-and-dns-names",
      "title": "IP Addresses and DNS names",
      "role": "content",
      "text": "Older versions of Sentinel did not support host names and required IP addresses to be specified everywhere.\nStarting with version 6.2, Sentinel has *optional* support for host names.\n\n**This capability is disabled by default. If you're going to enable DNS/hostnames support, please note:**\n\n1. The name resolution configuration on your Redis and Sentinel nodes must be reliable and be able to resolve addresses quickly. Unexpected delays in address resolution may have a negative impact on Sentinel.\n2. You should use hostnames everywhere and avoid mixing hostnames and IP addresses. To do that, use `replica-announce-ip <hostname>` and `sentinel announce-ip <hostname>` for all Redis and Sentinel instances, respectively.\n\nEnabling the `resolve-hostnames` global configuration allows Sentinel to accept host names:\n\n* As part of a `sentinel monitor` command\n* As a replica address, if the replica uses a host name value for `replica-announce-ip`\n\nSentinel will accept host names as valid inputs and resolve them, but will still refer to IP addresses when announcing an instance, updating configuration files, etc.\n\nEnabling the `announce-hostnames` global configuration makes Sentinel use host names instead. This affects replies to clients, values written in configuration files, the [`REPLICAOF`](https://redis.io/docs/latest/commands/replicaof) command issued to replicas, etc.\n\nThis behavior may not be compatible with all Sentinel clients, that may explicitly expect an IP address.\n\nUsing host names may be useful when clients use TLS to connect to instances and require a name rather than an IP address in order to perform certificate ASN matching."
    },
    {
      "id": "a-quick-tutorial",
      "title": "A quick tutorial",
      "role": "content",
      "text": "In the next sections of this document, all the details about [_Sentinel API_](#sentinel-api),\nconfiguration and semantics will be covered incrementally. However for people\nthat want to play with the system ASAP, this section is a tutorial that shows\nhow to configure and interact with 3 Sentinel instances.\n\nHere we assume that the instances are executed at port 5000, 5001, 5002.\nWe also assume that you have a running Redis master at port 6379 with a\nreplica running at port 6380. We will use the IPv4 loopback address 127.0.0.1\neverywhere during the tutorial, assuming you are running the simulation\non your personal computer.\n\nThe three Sentinel configuration files should look like the following:\n\n    port 5000\n    sentinel monitor mymaster 127.0.0.1 6379 2\n    sentinel down-after-milliseconds mymaster 5000\n    sentinel failover-timeout mymaster 60000\n    sentinel parallel-syncs mymaster 1\n\nThe other two configuration files will be identical but using 5001 and 5002\nas port numbers.\n\nA few things to note about the above configuration:\n\n* The master set is called `mymaster`. It identifies the master and its replicas. Since each *master set* has a different name, Sentinel can monitor different sets of masters and replicas at the same time.\n* The quorum was set to the value of 2 (last argument of `sentinel monitor` configuration directive).\n* The `down-after-milliseconds` value is 5000 milliseconds, that is 5 seconds, so masters will be detected as failing as soon as we don't receive any reply from our pings within this amount of time.\n\nOnce you start the three Sentinels, you'll see a few messages they log, like:\n\n    +monitor master mymaster 127.0.0.1 6379 quorum 2\n\nThis is a Sentinel event, and you can receive this kind of events via Pub/Sub\nif you [`SUBSCRIBE`](https://redis.io/docs/latest/commands/subscribe) to the event name as specified later in [_Pub/Sub Messages_ section](#pubsub-messages).\n\nSentinel generates and logs different events during failure detection and\nfailover.\n\nAsking Sentinel about the state of a master\n---\n\nThe most obvious thing to do with Sentinel to get started, is check if the\nmaster it is monitoring is doing well:\n\n    $ redis-cli -p 5000\n    127.0.0.1:5000> sentinel master mymaster\n     1) \"name\"\n     2) \"mymaster\"\n     3) \"ip\"\n     4) \"127.0.0.1\"\n     5) \"port\"\n     6) \"6379\"\n     7) \"runid\"\n     8) \"953ae6a589449c13ddefaee3538d356d287f509b\"\n     9) \"flags\"\n    10) \"master\"\n    11) \"link-pending-commands\"\n    12) \"0\"\n    13) \"link-refcount\"\n    14) \"1\"\n    15) \"last-ping-sent\"\n    16) \"0\"\n    17) \"last-ok-ping-reply\"\n    18) \"735\"\n    19) \"last-ping-reply\"\n    20) \"735\"\n    21) \"down-after-milliseconds\"\n    22) \"5000\"\n    23) \"info-refresh\"\n    24) \"126\"\n    25) \"role-reported\"\n    26) \"master\"\n    27) \"role-reported-time\"\n    28) \"532439\"\n    29) \"config-epoch\"\n    30) \"1\"\n    31) \"num-slaves\"\n    32) \"1\"\n    33) \"num-other-sentinels\"\n    34) \"2\"\n    35) \"quorum\"\n    36) \"2\"\n    37) \"failover-timeout\"\n    38) \"60000\"\n    39) \"parallel-syncs\"\n    40) \"1\"\n\nAs you can see, it prints a number of information about the master. There are\na few that are of particular interest for us:\n\n1. `num-other-sentinels` is 2, so we know the Sentinel already detected two more Sentinels for this master. If you check the logs you'll see the `+sentinel` events generated.\n2. `flags` is just `master`. If the master was down we could expect to see `s_down` or `o_down` flag as well here.\n3. `num-slaves` is correctly set to 1, so Sentinel also detected that there is an attached replica to our master.\n\nIn order to explore more about this instance, you may want to try the following\ntwo commands:\n\n    SENTINEL replicas mymaster\n    SENTINEL sentinels mymaster\n\nThe first will provide similar information about the replicas connected to the\nmaster, and the second about the other Sentinels.\n\nObtaining the address of the current master\n---\n\nAs we already specified, Sentinel also acts as a configuration provider for\nclients that want to connect to a set of master and replicas. Because of\npossible failovers or reconfigurations, clients have no idea about who is\nthe currently active master for a given set of instances, so Sentinel exports\nan API to ask this question:\n\n    127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster\n    1) \"127.0.0.1\"\n    2) \"6379\""
    },
    {
      "id": "testing-the-failover",
      "title": "Testing the failover",
      "role": "content",
      "text": "At this point our toy Sentinel deployment is ready to be tested. We can\njust kill our master and check if the configuration changes. To do so\nwe can just do:\n\n    redis-cli -p 6379 DEBUG sleep 30\n\nThis command will make our master no longer reachable, sleeping for 30 seconds.\nIt basically simulates a master hanging for some reason.\n\nIf you check the Sentinel logs, you should be able to see a lot of action:\n\n1. Each Sentinel detects the master is down with an `+sdown` event.\n2. This event is later escalated to `+odown`, which means that multiple Sentinels agree about the fact the master is not reachable.\n3. Sentinels vote a Sentinel that will start the first failover attempt.\n4. The failover happens.\n\nIf you ask again what is the current master address for `mymaster`, eventually\nwe should get a different reply this time:\n\n    127.0.0.1:5000> SENTINEL get-master-addr-by-name mymaster\n    1) \"127.0.0.1\"\n    2) \"6380\"\n\nSo far so good... At this point you may jump to create your Sentinel deployment\nor can read more to understand all the Sentinel commands and internals."
    },
    {
      "id": "sentinel-api",
      "title": "Sentinel API",
      "role": "content",
      "text": "Sentinel provides an API in order to inspect its state, check the health\nof monitored masters and replicas, subscribe in order to receive specific\nnotifications, and change the Sentinel configuration at run time.\n\nBy default Sentinel runs using TCP port 26379 (note that 6379 is the normal\nRedis port). Sentinels accept commands using the Redis protocol, so you can\nuse `redis-cli` or any other unmodified Redis client in order to talk with\nSentinel.\n\nIt is possible to directly query a Sentinel to check what is the state of\nthe monitored Redis instances from its point of view, to see what other\nSentinels it knows, and so forth. Alternatively, using Pub/Sub, it is possible\nto receive *push style* notifications from Sentinels, every time some event\nhappens, like a failover, or an instance entering an error condition, and\nso forth."
    },
    {
      "id": "sentinel-commands",
      "title": "Sentinel commands",
      "role": "content",
      "text": "The `SENTINEL` command is the main API for Sentinel. The following is the list of its subcommands (minimal version is noted for where applicable):\n\n* **SENTINEL CONFIG GET `<name>`** (`>= 6.2`) Get the current value of a global Sentinel configuration parameter. The specified name may be a wildcard, similar to the Redis [`CONFIG GET`](https://redis.io/docs/latest/commands/config-get) command.\n* **SENTINEL CONFIG SET `<name>` `<value>`** (`>= 6.2`) Set the value of a global Sentinel configuration parameter.\n* **SENTINEL CKQUORUM `<master name>`** Check if the current Sentinel configuration is able to reach the quorum needed to failover a master, and the majority needed to authorize the failover. This command should be used in monitoring systems to check if a Sentinel deployment is ok.\n* **SENTINEL FLUSHCONFIG** Force Sentinel to rewrite its configuration on disk, including the current Sentinel state. Normally Sentinel rewrites the configuration every time something changes in its state (in the context of the subset of the state which is persisted on disk across restart). However sometimes it is possible that the configuration file is lost because of operation errors, disk failures, package upgrade scripts or configuration managers. In those cases a way to force Sentinel to rewrite the configuration file is handy. This command works even if the previous configuration file is completely missing.\n* **SENTINEL FAILOVER `<master name>`** Force a failover as if the master was not reachable, and without asking for agreement to other Sentinels (however a new version of the configuration will be published so that the other Sentinels will update their configurations).\n* **SENTINEL GET-MASTER-ADDR-BY-NAME `<master name>`** Return the ip and port number of the master with that name. If a failover is in progress or terminated successfully for this master it returns the address and port of the promoted replica.\n* **SENTINEL INFO-CACHE** (`>= 3.2`) Return cached [`INFO`](https://redis.io/docs/latest/commands/info) output from masters and replicas.\n* **SENTINEL IS-MASTER-DOWN-BY-ADDR <ip> <port> <current-epoch> <runid>** Check if the master specified by ip:port is down from current Sentinel's point of view. This command is mostly for internal use.\n* **SENTINEL MASTER `<master name>`** Show the state and info of the specified master.\n* **SENTINEL MASTERS** Show a list of monitored masters and their state.\n* **SENTINEL MONITOR** Start Sentinel's monitoring. Refer to the [_Reconfiguring Sentinel at Runtime_ section](#reconfiguring-sentinel-at-runtime) for more information.\n* **SENTINEL MYID** (`>= 6.2`) Return the ID of the Sentinel instance.\n* **SENTINEL PENDING-SCRIPTS** This command returns information about pending scripts.\n* **SENTINEL REMOVE** Stop Sentinel's monitoring. Refer to the [_Reconfiguring Sentinel at Runtime_ section](#reconfiguring-sentinel-at-runtime) for more information.\n* **SENTINEL REPLICAS `<master name>`** (`>= 5.0`) Show a list of replicas for this master, and their state.\n* **SENTINEL SENTINELS `<master name>`** Show a list of sentinel instances for this master, and their state.\n* **SENTINEL SET** Set Sentinel's monitoring configuration. Refer to the [_Reconfiguring Sentinel at Runtime_ section](#reconfiguring-sentinel-at-runtime) for more information.\n* **SENTINEL SIMULATE-FAILURE (crash-after-election|crash-after-promotion|help)** (`>= 3.2`) This command simulates different Sentinel crash scenarios.\n* **SENTINEL RESET `<pattern>`** This command will reset all the masters with matching name. The pattern argument is a glob-style pattern. The reset process clears any previous state in a master (including a failover in progress), and removes every replica and sentinel already discovered and associated with the master.\n\nFor connection management and administration purposes, Sentinel supports the following subset of Redis' commands:\n\n* **ACL** (`>= 6.2`) This command manages the Sentinel Access Control List. For more information refer to the [ACL](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl) documentation page and the [_Sentinel Access Control List authentication_](#sentinel-access-control-list-authentication).\n* **AUTH** (`>= 5.0.1`) Authenticate a client connection. For more information refer to the [`AUTH`](https://redis.io/docs/latest/commands/auth) command and the [_Configuring Sentinel instances with authentication_ section](#configuring-sentinel-instances-with-authentication).\n* **CLIENT** This command manages client connections. For more information refer to its subcommands' pages.\n* **COMMAND** (`>= 6.2`) This command returns information about commands. For more information refer to the [`COMMAND`](https://redis.io/docs/latest/commands/command) command and its various subcommands.\n* **HELLO** (`>= 6.0`) Switch the connection's protocol. For more information refer to the [`HELLO`](https://redis.io/docs/latest/commands/hello) command.\n* **INFO** Return information and statistics about the Sentinel server. For more information see the [`INFO`](https://redis.io/docs/latest/commands/info) command.\n* **PING** This command simply returns PONG.\n* **ROLE** This command returns the string \"sentinel\" and a list of monitored masters. For more information refer to the [`ROLE`](https://redis.io/docs/latest/commands/role) command.\n* **SHUTDOWN** Shut down the Sentinel instance.\n\nLastly, Sentinel also supports the [`SUBSCRIBE`](https://redis.io/docs/latest/commands/subscribe), [`UNSUBSCRIBE`](https://redis.io/docs/latest/commands/unsubscribe), [`PSUBSCRIBE`](https://redis.io/docs/latest/commands/psubscribe) and [`PUNSUBSCRIBE`](https://redis.io/docs/latest/commands/punsubscribe) commands. Refer to the [_Pub/Sub Messages_ section](#pubsub-messages) for more details."
    },
    {
      "id": "reconfiguring-sentinel-at-runtime",
      "title": "Reconfiguring Sentinel at Runtime",
      "role": "content",
      "text": "Starting with Redis version 2.8.4, Sentinel provides an API in order to add, remove, or change the configuration of a given master. Note that if you have multiple sentinels you should apply the changes to all to your instances for Redis Sentinel to work properly. This means that changing the configuration of a single Sentinel does not automatically propagate the changes to the other Sentinels in the network.\n\nThe following is a list of `SENTINEL` subcommands used in order to update the configuration of a Sentinel instance.\n\n* **SENTINEL MONITOR `<name>` `<ip>` `<port>` `<quorum>`** This command tells the Sentinel to start monitoring a new master with the specified name, ip, port, and quorum. It is identical to the `sentinel monitor` configuration directive in `sentinel.conf` configuration file, with the difference that you can't use a hostname in as `ip`, but you need to provide an IPv4 or IPv6 address.\n* **SENTINEL REMOVE `<name>`** is used in order to remove the specified master: the master will no longer be monitored, and will totally be removed from the internal state of the Sentinel, so it will no longer listed by `SENTINEL masters` and so forth.\n* **SENTINEL SET `<name>` [`<option>` `<value>` ...]** The SET command is very similar to the [`CONFIG SET`](https://redis.io/docs/latest/commands/config-set) command of Redis, and is used in order to change configuration parameters of a specific master. Multiple option / value pairs can be specified (or none at all). All the configuration parameters that can be configured via `sentinel.conf` are also configurable using the SET command.\n\nThe following is an example of `SENTINEL SET` command in order to modify the `down-after-milliseconds` configuration of a master called `objects-cache`:\n\n    SENTINEL SET objects-cache-master down-after-milliseconds 1000\n\nAs already stated, `SENTINEL SET` can be used to set all the configuration parameters that are settable in the startup configuration file. Moreover it is possible to change just the master quorum configuration without removing and re-adding the master with `SENTINEL REMOVE` followed by `SENTINEL MONITOR`, but simply using:\n\n    SENTINEL SET objects-cache-master quorum 5\n\nNote that there is no equivalent GET command since `SENTINEL MASTER` provides all the configuration parameters in a simple to parse format (as a field/value pairs array).\n\nStarting with Redis version 6.2, Sentinel also allows getting and setting global configuration parameters which were only supported in the configuration file prior to that.\n\n* **SENTINEL CONFIG GET `<name>`** Get the current value of a global Sentinel configuration parameter. The specified name may be a wildcard, similar to the Redis [`CONFIG GET`](https://redis.io/docs/latest/commands/config-get) command.\n* **SENTINEL CONFIG SET `<name>` `<value>`** Set the value of a global Sentinel configuration parameter.\n\nGlobal parameters that can be manipulated include:\n\n* `resolve-hostnames`, `announce-hostnames`. See [_IP addresses and DNS names_](#ip-addresses-and-dns-names).\n* `announce-ip`, `announce-port`. See [_Sentinel, Docker, NAT, and possible issues_](#sentinel-docker-nat-and-possible-issues).\n* `sentinel-user`, `sentinel-pass`. See [_Configuring Sentinel instances with authentication_](#configuring-sentinel-instances-with-authentication)."
    },
    {
      "id": "adding-or-removing-sentinels",
      "title": "Adding or removing Sentinels",
      "role": "content",
      "text": "Adding a new Sentinel to your deployment is a simple process because of the\nauto-discover mechanism implemented by Sentinel. All you need to do is to\nstart the new Sentinel configured to monitor the currently active master.\nWithin 10 seconds the Sentinel will acquire the list of other Sentinels and\nthe set of replicas attached to the master.\n\nIf you need to add multiple Sentinels at once, it is suggested to add it\none after the other, waiting for all the other Sentinels to already know\nabout the first one before adding the next. This is useful in order to still\nguarantee that majority can be achieved only in one side of a partition,\nin the chance failures should happen in the process of adding new Sentinels.\n\nThis can be easily achieved by adding every new Sentinel with a 30 seconds delay, and during absence of network partitions.\n\nAt the end of the process it is possible to use the command\n`SENTINEL MASTER mastername` in order to check if all the Sentinels agree about\nthe total number of Sentinels monitoring the master.\n\nRemoving a Sentinel is a bit more complex: **Sentinels never forget already seen\nSentinels**, even if they are not reachable for a long time, since we don't\nwant to dynamically change the majority needed to authorize a failover and\nthe creation of a new configuration number. So in order to remove a Sentinel\nthe following steps should be performed in absence of network partitions:\n\n1. Stop the Sentinel process of the Sentinel you want to remove.\n2. Send a `SENTINEL RESET *` command to all the other Sentinel instances (instead of `*` you can use the exact master name if you want to reset just a single master). One after the other, waiting at least 30 seconds between instances.\n3. Check that all the Sentinels agree about the number of Sentinels currently active, by inspecting the output of `SENTINEL MASTER mastername` of every Sentinel."
    },
    {
      "id": "removing-the-old-master-or-unreachable-replicas",
      "title": "Removing the old master or unreachable replicas",
      "role": "content",
      "text": "Sentinels never forget about replicas of a given master, even when they are\nunreachable for a long time. This is useful, because Sentinels should be able\nto correctly reconfigure a returning replica after a network partition or a\nfailure event.\n\nMoreover, after a failover, the failed over master is virtually added as a\nreplica of the new master, this way it will be reconfigured to replicate with\nthe new master as soon as it will be available again.\n\nHowever sometimes you want to remove a replica (that may be the old master)\nforever from the list of replicas monitored by Sentinels.\n\nIn order to do this, you need to send a `SENTINEL RESET mastername` command\nto all the Sentinels: they'll refresh the list of replicas within the next\n10 seconds, only adding the ones listed as correctly replicating from the\ncurrent master [`INFO`](https://redis.io/docs/latest/commands/info) output."
    },
    {
      "id": "pubsub-messages",
      "title": "Pub/Sub messages",
      "role": "content",
      "text": "A client can use a Sentinel as a Redis-compatible Pub/Sub server\n(but you can't use [`PUBLISH`](https://redis.io/docs/latest/commands/publish)) in order to [`SUBSCRIBE`](https://redis.io/docs/latest/commands/subscribe) or [`PSUBSCRIBE`](https://redis.io/docs/latest/commands/psubscribe) to\nchannels and get notified about specific events.\n\nThe channel name is the same as the name of the event. For instance the\nchannel named `+sdown` will receive all the notifications related to instances\nentering an `SDOWN` (SDOWN means the instance is no longer reachable from\nthe point of view of the Sentinel you are querying) condition.\n\nTo get all the messages simply subscribe using `PSUBSCRIBE *`.\n\nThe following is a list of channels and message formats you can receive using\nthis API. The first word is the channel / event name, the rest is the format of the data.\n\nNote: where *instance details* is specified it means that the following arguments are provided to identify the target instance:\n\n    <instance-type> <name> <ip> <port> @ <master-name> <master-ip> <master-port>\n\nThe part identifying the master (from the @ argument to the end) is optional\nand is only specified if the instance is not a master itself.\n\n* **+reset-master** `<instance details>` -- The master was reset.\n* **+slave** `<instance details>` -- A new replica was detected and attached.\n* **+failover-state-reconf-slaves** `<instance details>` -- Failover state changed to `reconf-slaves` state.\n* **+failover-detected** `<instance details>` -- A failover started by another Sentinel or any other external entity was detected (An attached replica turned into a master).\n* **+slave-reconf-sent** `<instance details>` -- The leader sentinel sent the [`REPLICAOF`](https://redis.io/docs/latest/commands/replicaof) command to this instance in order to reconfigure it for the new replica.\n* **+slave-reconf-inprog** `<instance details>` -- The replica being reconfigured showed to be a replica of the new master ip:port pair, but the synchronization process is not yet complete.\n* **+slave-reconf-done** `<instance details>` -- The replica is now synchronized with the new master.\n* **-dup-sentinel** `<instance details>` -- One or more sentinels for the specified master were removed as duplicated (this happens for instance when a Sentinel instance is restarted).\n* **+sentinel** `<instance details>` -- A new sentinel for this master was detected and attached.\n* **+sdown** `<instance details>` -- The specified instance is now in Subjectively Down state.\n* **-sdown** `<instance details>` -- The specified instance is no longer in Subjectively Down state.\n* **+odown** `<instance details>` -- The specified instance is now in Objectively Down state.\n* **-odown** `<instance details>` -- The specified instance is no longer in Objectively Down state.\n* **+new-epoch** `<instance details>` -- The current epoch was updated.\n* **+try-failover** `<instance details>` -- New failover in progress, waiting to be elected by the majority.\n* **+elected-leader** `<instance details>` -- Won the election for the specified epoch, can do the failover.\n* **+failover-state-select-slave** `<instance details>` -- New failover state is `select-slave`: we are trying to find a suitable replica for promotion.\n* **no-good-slave** `<instance details>` -- There is no good replica to promote. Currently we'll try after some time, but probably this will change and the state machine will abort the failover at all in this case.\n* **selected-slave** `<instance details>` -- We found the specified good replica to promote.\n* **failover-state-send-slaveof-noone** `<instance details>` -- We are trying to reconfigure the promoted replica as master, waiting for it to switch.\n* **failover-end-for-timeout** `<instance details>` -- The failover terminated for timeout, replicas will eventually be configured to replicate with the new master anyway.\n* **failover-end** `<instance details>` -- The failover terminated with success. All the replicas appears to be reconfigured to replicate with the new master.\n* **switch-master** `<master name> <oldip> <oldport> <newip> <newport>` -- The master new IP and address is the specified one after a configuration change. This is **the message most external users are interested in**.\n* **+tilt** -- Tilt mode entered.\n* **-tilt** -- Tilt mode exited."
    },
    {
      "id": "handling-of--busy-state",
      "title": "Handling of -BUSY state",
      "role": "content",
      "text": "The -BUSY error is returned by a Redis instance when a Lua script is running for\nmore time than the configured Lua script time limit. When this happens before\ntriggering a fail over Redis Sentinel will try to send a [`SCRIPT KILL`](https://redis.io/docs/latest/commands/script-kill)\ncommand, that will only succeed if the script was read-only.\n\nIf the instance is still in an error condition after this try, it will\neventually be failed over.\n\nReplicas priority\n---\n\nRedis instances have a configuration parameter called `replica-priority`.\nThis information is exposed by Redis replica instances in their [`INFO`](https://redis.io/docs/latest/commands/info) output,\nand Sentinel uses it in order to pick a replica among the ones that can be\nused in order to failover a master:\n\n1. If the replica priority is set to 0, the replica is never promoted to master.\n2. Replicas with a *lower* priority number are preferred by Sentinel.\n\nFor example if there is a replica S1 in the same data center of the current\nmaster, and another replica S2 in another data center, it is possible to set\nS1 with a priority of 10 and S2 with a priority of 100, so that if the master\nfails and both S1 and S2 are available, S1 will be preferred.\n\nFor more information about the way replicas are selected, please check the [_Replica selection and priority_ section](#replica-selection-and-priority) of this documentation."
    },
    {
      "id": "sentinel-and-redis-authentication",
      "title": "Sentinel and Redis authentication",
      "role": "content",
      "text": "When the master is configured to require authentication from clients,\nas a security measure, replicas need to also be aware of the credentials in\norder to authenticate with the master and create the master-replica connection\nused for the asynchronous replication protocol."
    },
    {
      "id": "redis-access-control-list-authentication",
      "title": "Redis Access Control List authentication",
      "role": "content",
      "text": "Starting with Redis 6, user authentication and permission is managed with the [Access Control List (ACL)](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl).\n\nIn order for Sentinels to connect to Redis server instances when they are\nconfigured with ACL, the Sentinel configuration must include the\nfollowing directives:\n\n    sentinel auth-user <master-name> <username>\n    sentinel auth-pass <master-name> <password>\n\nWhere `<username>` and `<password>` are the username and password for accessing the group's instances. These credentials should be provisioned on all of the group's Redis instances with the minimal control permissions. For example:\n\n    127.0.0.1:6379> ACL SETUSER sentinel-user ON >somepassword resetchannels &__sentinel__:hello +multi +slaveof +ping +exec +subscribe +config|rewrite +role +publish +info +client|setname +client|kill +script|kill\n\nThe only Pub/Sub channel that Sentinel uses on a monitored node is `__sentinel__:hello`, so\nthis user is granted access to that channel only (`resetchannels &__sentinel__:hello`) rather\nthan to all channels (`allchannels`, equivalently `&*`).\n\n> [!WARNING]\n> Sentinel discovery relies on the reserved `__sentinel__:hello` Pub/Sub channel on each\n> monitored master and replica. Sentinels periodically publish their presence there and\n> subscribe to it to discover other Sentinels and learn the current topology. These messages\n> carry no authentication and are trusted by receiving Sentinels, so any client that can\n> publish to this channel on a monitored node can send forged HELLO messages that redirect a Sentinel to an attacker-controlled endpoint,\n> potentially exposing the configured master credentials sent over command and Pub/Sub connections, and\n> trigger spurious failovers or a denial of service.\n>\n> Because of this, do not grant broad Pub/Sub access on Sentinel-monitored Redis nodes:\n>\n> * No user needs access to all channels (`allchannels` / `&*`). The Sentinel `auth-user`\n>   needs access only to the `__sentinel__:hello` channel (`&__sentinel__:hello`), and no\n>   other user should be able to publish to or subscribe to it. Since Redis 7.0, the default\n>   value of `acl-pubsub-default` is `resetchannels`, so newly created ACL users have no\n>   channel access unless you grant it explicitly. Be especially careful with the legacy\n>   `default` user, which may still have broad access. This documentation change does not alter existing ACL configurations. Review ACL users on monitored masters and replicas,\n>   including the legacy `default` user, and confirm that no non-Sentinel user can publish to or subscribe to\n>   `__sentinel__:hello`.\n> * Treat the `__sentinel__:` channel prefix as reserved for Sentinel's internal use."
    },
    {
      "id": "redis-password-only-authentication",
      "title": "Redis password-only authentication",
      "role": "content",
      "text": "Until Redis 6, authentication is achieved using the following configuration directives:\n\n* `requirepass` in the master, in order to set the authentication password, and to make sure the instance will not process requests for non authenticated clients.\n* `masterauth` in the replicas in order for the replicas to authenticate with the master in order to correctly replicate data from it.\n\nWhen Sentinel is used, there is not a single master, since after a failover\nreplicas may play the role of masters, and old masters can be reconfigured in\norder to act as replicas, so what you want to do is to set the above directives\nin all your instances, both masters and replicas.\n\nThis is also usually a sane setup since you don't want to protect\ndata only in the master, having the same data accessible in the replicas.\n\nHowever, in the uncommon case where you need a replica that is accessible\nwithout authentication, you can still do it by setting up **a replica priority\nof zero**, to prevent this replica from being promoted to master, and\nconfiguring in this replica only the `masterauth` directive, without\nusing the `requirepass` directive, so that data will be readable by\nunauthenticated clients.\n\nIn order for Sentinels to connect to Redis server instances when they are\nconfigured with `requirepass`, the Sentinel configuration must include the\n`sentinel auth-pass` directive, in the format:\n\n    sentinel auth-pass <master-name> <password>\n\nConfiguring Sentinel instances with authentication\n---\n\nSentinel instances themselves can be secured by requiring clients to authenticate via the [`AUTH`](https://redis.io/docs/latest/commands/auth) command. Starting with Redis 6.2, the [Access Control List (ACL)](https://redis.io/docs/latest/operate/oss_and_stack/management/security/acl) is available, whereas previous versions (starting with Redis 5.0.1) support password-only authentication. \n\nNote that Sentinel's authentication configuration should be **applied to each of the instances** in your deployment, and **all instances should use the same configuration**. Furthermore, ACL and password-only authentication should not be used together."
    },
    {
      "id": "sentinel-access-control-list-authentication",
      "title": "Sentinel Access Control List authentication",
      "role": "content",
      "text": "The first step in securing a Sentinel instance with ACL is preventing any unauthorized access to it. To do that, you'll need to disable the default superuser (or at the very least set it up with a strong password) and create a new one and allow it access to Pub/Sub channels:\n\n    127.0.0.1:5000> ACL SETUSER admin ON >admin-password allchannels +@all\n    OK\n    127.0.0.1:5000> ACL SETUSER default off\n    OK\n\nThe default user is used by Sentinel to connect to other instances. You can provide the credentials of another superuser with the following configuration directives:\n\n    sentinel sentinel-user <username>\n    sentinel sentinel-pass <password>\n\nWhere `<username>` and `<password>` are the Sentinel's superuser and password, respectively (e.g. `admin` and `admin-password` in the example above).\n\nLastly, for authenticating incoming client connections, you can create a Sentinel restricted user profile such as the following:\n\n    127.0.0.1:5000> ACL SETUSER sentinel-user ON >user-password -@all +auth +client|getname +client|id +client|setname +command +hello +ping +role +sentinel|get-master-addr-by-name +sentinel|master +sentinel|myid +sentinel|replicas +sentinel|sentinels +sentinel|masters\n\nRefer to the documentation of your Sentinel client of choice for further information."
    },
    {
      "id": "sentinel-password-only-authentication",
      "title": "Sentinel password-only authentication",
      "role": "content",
      "text": "To use Sentinel with password-only authentication, add the `requirepass` configuration directive to **all** your Sentinel instances as follows:\n\n    requirepass \"your_password_here\"\n\nWhen configured this way, Sentinels will do two things:\n\n1. A password will be required from clients in order to send commands to Sentinels. This is obvious since this is how such configuration directive works in Redis in general.\n2. Moreover the same password configured to access the local Sentinel, will be used by this Sentinel instance in order to authenticate to all the other Sentinel instances it connects to.\n\nThis means that **you will have to configure the same `requirepass` password in all the Sentinel instances**. This way every Sentinel can talk with every other Sentinel without any need to configure for each Sentinel the password to access all the other Sentinels, that would be very impractical.\n\nBefore using this configuration, make sure your client library can send the [`AUTH`](https://redis.io/docs/latest/commands/auth) command to Sentinel instances."
    },
    {
      "id": "sentinel-clients-implementation",
      "title": "Sentinel clients implementation",
      "role": "content",
      "text": "---\n\nSentinel requires explicit client support, unless the system is configured to execute a script that performs a transparent redirection of all the requests to the new master instance (virtual IP or other similar systems). The topic of client libraries implementation is covered in the document [Sentinel clients guidelines](https://redis.io/docs/latest/develop/reference/sentinel-clients)."
    },
    {
      "id": "more-advanced-concepts",
      "title": "More advanced concepts",
      "role": "content",
      "text": "In the following sections we'll cover a few details about how Sentinel works,\nwithout resorting to implementation details and algorithms that will be\ncovered in the final part of this document."
    },
    {
      "id": "sdown-and-odown-failure-state",
      "title": "SDOWN and ODOWN failure state",
      "role": "content",
      "text": "Redis Sentinel has two different concepts of *being down*, one is called\na *Subjectively Down* condition (SDOWN) and is a down condition that is\nlocal to a given Sentinel instance. Another is called *Objectively Down*\ncondition (ODOWN) and is reached when enough Sentinels (at least the\nnumber configured as the `quorum` parameter of the monitored master) have\nan SDOWN condition, and get feedback from other Sentinels using\nthe `SENTINEL is-master-down-by-addr` command.\n\nFrom the point of view of a Sentinel an SDOWN condition is reached when it\ndoes not receive a valid reply to PING requests for the number of seconds\nspecified in the configuration as `is-master-down-after-milliseconds`\nparameter.\n\nAn acceptable reply to PING is one of the following:\n\n* PING replied with +PONG.\n* PING replied with -LOADING error.\n* PING replied with -MASTERDOWN error.\n\nAny other reply (or no reply at all) is considered non valid.\nHowever note that **a logical master that advertises itself as a replica in\nthe INFO output is considered to be down**.\n\nNote that SDOWN requires that no acceptable reply is received for the whole\ninterval configured, so for instance if the interval is 30000 milliseconds\n(30 seconds) and we receive an acceptable ping reply every 29 seconds, the\ninstance is considered to be working.\n\nSDOWN is not enough to trigger a failover: it only means a single Sentinel\nbelieves a Redis instance is not available. To trigger a failover, the\nODOWN state must be reached.\n\nTo switch from SDOWN to ODOWN no strong consensus algorithm is used, but\njust a form of gossip: if a given Sentinel gets reports that a master\nis not working from enough Sentinels **in a given time range**, the SDOWN is\npromoted to ODOWN. If this acknowledge is later missing, the flag is cleared.\n\nA more strict authorization that uses an actual majority is required in\norder to really start the failover, but no failover can be triggered without\nreaching the ODOWN state.\n\nThe ODOWN condition **only applies to masters**. For other kind of instances\nSentinel doesn't require to act, so the ODOWN state is never reached for replicas\nand other sentinels, but only SDOWN is.\n\nHowever SDOWN has also semantic implications. For example a replica in SDOWN\nstate is not selected to be promoted by a Sentinel performing a failover.\n\nSentinels and replicas auto discovery\n---\n\nSentinels stay connected with other Sentinels in order to reciprocally\ncheck the availability of each other, and to exchange messages. However you\ndon't need to configure a list of other Sentinel addresses in every Sentinel\ninstance you run, as Sentinel uses the Redis instances Pub/Sub capabilities\nin order to discover the other Sentinels that are monitoring the same masters\nand replicas.\n\nThis feature is implemented by sending *hello messages* into the channel named\n`__sentinel__:hello`.\n\nSimilarly you don't need to configure what is the list of the replicas attached\nto a master, as Sentinel will auto discover this list querying Redis.\n\n* Every Sentinel publishes a message to every monitored master and replica Pub/Sub channel `__sentinel__:hello`, every two seconds, announcing its presence with ip, port, runid.\n* Every Sentinel is subscribed to the Pub/Sub channel `__sentinel__:hello` of every master and replica, looking for unknown sentinels. When new sentinels are detected, they are added as sentinels of this master.\n* Hello messages also include the full current configuration of the master. If the receiving Sentinel has a configuration for a given master which is older than the one received, it updates to the new configuration immediately.\n* Before adding a new sentinel to a master a Sentinel always checks if there is already a sentinel with the same runid or the same address (ip and port pair). In that case all the matching sentinels are removed, and the new added.\n\nSentinel reconfiguration of instances outside the failover procedure\n---\n\nEven when no failover is in progress, Sentinels will always try to set the\ncurrent configuration on monitored instances. Specifically:\n\n* Replicas (according to the current configuration) that claim to be masters, will be configured as replicas to replicate with the current master.\n* Replicas connected to a wrong master, will be reconfigured to replicate with the right master.\n\nFor Sentinels to reconfigure replicas, the wrong configuration must be observed for some time, that is greater than the period used to broadcast new configurations.\n\nThis prevents Sentinels with a stale configuration (for example because they just rejoined from a partition) will try to change the replicas configuration before receiving an update.\n\nAlso note how the semantics of always trying to impose the current configuration makes the failover more resistant to partitions:\n\n* Masters failed over are reconfigured as replicas when they return available.\n* Replicas partitioned away during a partition are reconfigured once reachable.\n\nThe important lesson to remember about this section is: **Sentinel is a system where each process will always try to impose the last logical configuration to the set of monitored instances**."
    },
    {
      "id": "replica-selection-and-priority",
      "title": "Replica selection and priority",
      "role": "content",
      "text": "When a Sentinel instance is ready to perform a failover, since the master\nis in `ODOWN` state and the Sentinel received the authorization to failover\nfrom the majority of the Sentinel instances known, a suitable replica needs\nto be selected.\n\nThe replica selection process evaluates the following information about replicas:\n\n1. Disconnection time from the master.\n2. Replica priority.\n3. Replication offset processed.\n4. Run ID.\n\nA replica that is found to be disconnected from the master for more than ten\ntimes the configured master timeout (down-after-milliseconds option), plus\nthe time the master is also not available from the point of view of the\nSentinel doing the failover, is considered to be not suitable for the failover\nand is skipped.\n\nIn more rigorous terms, a replica whose the [`INFO`](https://redis.io/docs/latest/commands/info) output suggests it has been\ndisconnected from the master for more than:\n\n    (down-after-milliseconds * 10) + milliseconds_since_master_is_in_SDOWN_state\n\nIs considered to be unreliable and is disregarded entirely.\n\nThe replica selection only considers the replicas that passed the above test,\nand sorts it based on the above criteria, in the following order.\n\n1. The replicas are sorted by `replica-priority` as configured in the `redis.conf` file of the Redis instance. A lower priority will be preferred.\n2. If the priority is the same, the replication offset processed by the replica is checked, and the replica that received more data from the master is selected.\n3. If multiple replicas have the same priority and processed the same data from the master, a further check is performed, selecting the replica with the lexicographically smaller run ID. Having a lower run ID is not a real advantage for a replica, but is useful in order to make the process of replica selection more deterministic, instead of resorting to select a random replica.\n\nIn most cases, `replica-priority` does not need to be set explicitly so all\ninstances will use the same default value. If there is a particular fail-over\npreference, `replica-priority` must be set on all instances, including masters,\nas a master may become a replica at some future point in time - and it will then\nneed the proper `replica-priority` settings.\n\nA Redis instance can be configured with a special `replica-priority` of zero\nin order to be **never selected** by Sentinels as the new master.\nHowever a replica configured in this way will still be reconfigured by\nSentinels in order to replicate with the new master after a failover, the\nonly difference is that it will never become a master itself."
    },
    {
      "id": "algorithms-and-internals",
      "title": "Algorithms and internals",
      "role": "content",
      "text": "In the following sections we will explore the details of Sentinel behavior.\nIt is not strictly needed for users to be aware of all the details, but a\ndeep understanding of Sentinel may help to deploy and operate Sentinel in\na more effective way."
    },
    {
      "id": "quorum",
      "title": "Quorum",
      "role": "content",
      "text": "The previous sections showed that every master monitored by Sentinel is associated to a configured **quorum**. It specifies the number of Sentinel processes\nthat need to agree about the unreachability or error condition of the master in\norder to trigger a failover.\n\nHowever, after the failover is triggered, in order for the failover to actually be performed, **at least a majority of Sentinels must authorize the Sentinel to\nfailover**. Sentinel never performs a failover in the partition where a\nminority of Sentinels exist.\n\nLet's try to make things a bit more clear:\n\n* Quorum: the number of Sentinel processes that need to detect an error condition in order for a master to be flagged as **ODOWN**.\n* The failover is triggered by the **ODOWN** state.\n* Once the failover is triggered, the Sentinel trying to failover is required to ask for authorization to a majority of Sentinels (or more than the majority if the quorum is set to a number greater than the majority).\n\nThe difference may seem subtle but is actually quite simple to understand and use.  For example if you have 5 Sentinel instances, and the quorum is set to 2, a failover will be triggered as soon as 2 Sentinels believe that the master is not reachable, however one of the two Sentinels will be able to failover only if it gets authorization at least from 3 Sentinels.\n\nIf instead the quorum is configured to 5, all the Sentinels must agree about the master error condition, and the authorization from all Sentinels is required in order to failover.\n\nThis means that the quorum can be used to tune Sentinel in two ways:\n\n1. If a quorum is set to a value smaller than the majority of Sentinels we deploy, we are basically making Sentinel more sensitive to master failures, triggering a failover as soon as even just a minority of Sentinels is no longer able to talk with the master.\n2. If a quorum is set to a value greater than the majority of Sentinels, we are making Sentinel able to failover only when there are a very large number (larger than majority) of well connected Sentinels which agree about the master being down."
    },
    {
      "id": "configuration-epochs",
      "title": "Configuration epochs",
      "role": "configuration",
      "text": "Sentinels require to get authorizations from a majority in order to start a\nfailover for a few important reasons:\n\nWhen a Sentinel is authorized, it gets a unique **configuration epoch** for the master it is failing over. This is a number that will be used to version the new configuration after the failover is completed. Because a majority agreed that a given version was assigned to a given Sentinel, no other Sentinel will be able to use it. This means that every configuration of every failover is versioned with a unique version. We'll see why this is so important.\n\nMoreover Sentinels have a rule: if a Sentinel voted another Sentinel for the failover of a given master, it will wait some time to try to failover the same master again. This delay is the `2 * failover-timeout` you can configure in `sentinel.conf`. This means that Sentinels will not try to failover the same master at the same time, the first to ask to be authorized will try, if it fails another will try after some time, and so forth.\n\nRedis Sentinel guarantees the *liveness* property that if a majority of Sentinels are able to talk, eventually one will be authorized to failover if the master is down.\n\nRedis Sentinel also guarantees the *safety* property that every Sentinel will failover the same master using a different *configuration epoch*."
    },
    {
      "id": "configuration-propagation",
      "title": "Configuration propagation",
      "role": "configuration",
      "text": "Once a Sentinel is able to failover a master successfully, it will start to broadcast the new configuration so that the other Sentinels will update their information about a given master.\n\nFor a failover to be considered successful, it requires that the Sentinel was able to send the `REPLICAOF NO ONE` command to the selected replica, and that the switch to master was later observed in the [`INFO`](https://redis.io/docs/latest/commands/info) output of the master.\n\nAt this point, even if the reconfiguration of the replicas is in progress, the failover is considered to be successful, and all the Sentinels are required to start reporting the new configuration.\n\nThe way a new configuration is propagated is the reason why we need that every\nSentinel failover is authorized with a different version number (configuration epoch).\n\nEvery Sentinel continuously broadcast its version of the configuration of a master using Redis Pub/Sub messages, both in the master and all the replicas.  At the same time all the Sentinels wait for messages to see what is the configuration\nadvertised by the other Sentinels.\n\nConfigurations are broadcast in the `__sentinel__:hello` Pub/Sub channel.\n\nBecause every configuration has a different version number, the greater version\nalways wins over smaller versions.\n\nSo for example the configuration for the master `mymaster` start with all the\nSentinels believing the master is at 192.168.1.50:6379. This configuration\nhas version 1. After some time a Sentinel is authorized to failover with version 2. If the failover is successful, it will start to broadcast a new configuration, let's say 192.168.1.50:9000, with version 2. All the other instances will see this configuration and will update their configuration accordingly, since the new configuration has a greater version.\n\nThis means that Sentinel guarantees a second liveness property: a set of\nSentinels that are able to communicate will all converge to the same configuration with the higher version number.\n\nBasically if the net is partitioned, every partition will converge to the higher\nlocal configuration. In the special case of no partitions, there is a single\npartition and every Sentinel will agree about the configuration."
    },
    {
      "id": "consistency-under-partitions",
      "title": "Consistency under partitions",
      "role": "content",
      "text": "Redis Sentinel configurations are eventually consistent, so every partition will\nconverge to the higher configuration available.\nHowever in a real-world system using Sentinel there are three different players:\n\n* Redis instances.\n* Sentinel instances.\n* Clients.\n\nIn order to define the behavior of the system we have to consider all three.\n\nThe following is a simple network where there are 3 nodes, each running\na Redis instance, and a Sentinel instance:\n\n                +-------------+\n                | Sentinel 1  |----- Client A\n                | Redis 1 (M) |\n                +-------------+\n                        |\n                        |\n    +-------------+     |          +------------+\n    | Sentinel 2  |-----+-- // ----| Sentinel 3 |----- Client B\n    | Redis 2 (S) |                | Redis 3 (M)|\n    +-------------+                +------------+\n\nIn this system the original state was that Redis 3 was the master, while\nRedis 1 and 2 were replicas. A partition occurred isolating the old master.\nSentinels 1 and 2 started a failover promoting Sentinel 1 as the new master.\n\nThe Sentinel properties guarantee that Sentinel 1 and 2 now have the new\nconfiguration for the master. However Sentinel 3 has still the old configuration\nsince it lives in a different partition.\n\nWe know that Sentinel 3 will get its configuration updated when the network\npartition will heal, however what happens during the partition if there\nare clients partitioned with the old master?\n\nClients will be still able to write to Redis 3, the old master. When the\npartition will rejoin, Redis 3 will be turned into a replica of Redis 1, and\nall the data written during the partition will be lost.\n\nDepending on your configuration you may want or not that this scenario happens:\n\n* If you are using Redis as a cache, it could be handy that Client B is still able to write to the old master, even if its data will be lost.\n* If you are using Redis as a store, this is not good and you need to configure the system in order to partially prevent this problem.\n\nSince Redis is asynchronously replicated, there is no way to totally prevent data loss in this scenario, however you can bound the divergence between Redis 3 and Redis 1\nusing the following Redis configuration option:\n\n    min-replicas-to-write 1\n    min-replicas-max-lag 10\n\nWith the above configuration (please see the self-commented `redis.conf` example in the Redis distribution for more information) a Redis instance, when acting as a master, will stop accepting writes if it can't write to at least 1 replica. Since replication is asynchronous *not being able to write* actually means that the replica is either disconnected, or is not sending us asynchronous acknowledges for more than the specified `max-lag` number of seconds.\n\nUsing this configuration the Redis 3 in the above example will become unavailable after 10 seconds. When the partition heals, the Sentinel 3 configuration will converge to\nthe new one, and Client B will be able to fetch a valid configuration and continue.\n\nIn general Redis + Sentinel as a whole are an **eventually consistent system** where the merge function is **last failover wins**, and the data from old masters are discarded to replicate the data of the current master, so there is always a window for losing acknowledged writes. This is due to Redis asynchronous\nreplication and the discarding nature of the \"virtual\" merge function of the system. Note that this is not a limitation of Sentinel itself, and if you orchestrate the failover with a strongly consistent replicated state machine, the same properties will still apply. There are only two ways to avoid losing acknowledged writes:\n\n1. Use synchronous replication (and a proper consensus algorithm to run a replicated state machine).\n2. Use an eventually consistent system where different versions of the same object can be merged.\n\nRedis currently is not able to use any of the above systems, and is currently outside the development goals. However there are proxies implementing solution \"2\" on top of Redis stores such as SoundCloud [Roshi](https://github.com/soundcloud/roshi), or Netflix [Dynomite](https://github.com/Netflix/dynomite).\n\nSentinel persistent state\n---\n\nSentinel state is persisted in the sentinel configuration file. For example\nevery time a new configuration is received, or created (leader Sentinels), for\na master, the configuration is persisted on disk together with the configuration\nepoch. This means that it is safe to stop and restart Sentinel processes."
    },
    {
      "id": "tilt-mode",
      "title": "TILT mode",
      "role": "content",
      "text": "Redis Sentinel is heavily dependent on the computer time: for instance in\norder to understand if an instance is available it remembers the time of the\nlatest successful reply to the PING command, and compares it with the current\ntime to understand how old it is.\n\nHowever if the computer time changes in an unexpected way, or if the computer\nis very busy, or the process blocked for some reason, Sentinel may start to\nbehave in an unexpected way.\n\nThe TILT mode is a special \"protection\" mode that a Sentinel can enter when\nsomething odd is detected that can lower the reliability of the system.\nThe Sentinel timer interrupt is normally called 10 times per second, so we\nexpect that more or less 100 milliseconds will elapse between two calls\nto the timer interrupt.\n\nWhat a Sentinel does is to register the previous time the timer interrupt\nwas called, and compare it with the current call: if the time difference\nis negative or unexpectedly big (2 seconds or more) the TILT mode is entered\n(or if it was already entered the exit from the TILT mode postponed).\n\nWhen in TILT mode the Sentinel will continue to monitor everything, but:\n\n* It stops acting at all.\n* It starts to reply negatively to `SENTINEL is-master-down-by-addr` requests as the ability to detect a failure is no longer trusted.\n\nIf everything appears to be normal for 30 second, the TILT mode is exited.\n \nIn the Sentinel TILT mode, if we send the INFO command, we could get the following response:\n\n    $ redis-cli -p 26379\n    127.0.0.1:26379> info\n    (Other information from Sentinel server skipped.)\n\n    # Sentinel\n    sentinel_masters:1\n    sentinel_tilt:0\n    sentinel_tilt_since_seconds:-1\n    sentinel_running_scripts:0\n    sentinel_scripts_queue_length:0\n    sentinel_simulate_failure_flags:0\n    master0:name=mymaster,status=ok,address=127.0.0.1:6379,slaves=0,sentinels=1\n\nThe field \"sentinel_tilt_since_seconds\" indicates how many seconds the Sentinel already is in the TILT mode.\nIf it is not in TILT mode, the value will be -1.\n\nNote that in some ways TILT mode could be replaced using the monotonic clock\nAPI that many kernels offer. However it is not still clear if this is a good\nsolution since the current system avoids issues in case the process is just\nsuspended or not executed by the scheduler for a long time.\n\n**A note about the word slave used in this man page**: Starting with Redis 5, if not for backward compatibility, the Redis project no longer uses the word slave. Unfortunately in this command the word slave is part of the protocol, so we'll be able to remove such occurrences only when this API will be naturally deprecated."
    }
  ],
  "examples": []
}